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EXECUTIVE SUMMARY 



The development of a Quality Control (QC) system for the Pell Grant Program is 
a complex undertaking. This paper proposes a preliminary design for a QC system for 
the Pell Grant Program, based on a functional analysis of QC needs in the Office of 
Student Financial Assistance (OSFA). The applicability of the general design is also 
considered for other student aid programs administered by OSFA. 

The conceptual framework for this project utilized a strategic approach to QC 
system design and testing. The basic steps implied in the strategic approach are: 

• Identify major sources of error. 

• Conduct a functional analysis of the current delivery system. 

• Conceptualize QC system modular components, 

• Select and prioritize modular components. 

• Proceed with development of selected components. 

• Perform system tests on modular components as they are developed. 

This report represents the second and thirr* teps of this process. The functional 

analysis of the Pell delivery system empto ed documenting and analyzing current 

sources of information and management procedures, since the most viable 
configuration option appears to be a combined automated/manual system that uses 
existing data. This emphasis increases the possibility that the QC system would be 
resource saving using and enhancing existing data and management practices, rather 
than resource draining, placing new demands on the current delivery system. The 
analytic framework used to develop the preliminary QC system design considered QC 
information requirements from three perspectives: 

• Strategic policy : Refers to the policy and political arena and the types of 
quality control information, reporting, monitoring, and analysis required 
for improved policymaking. 

iii 
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• Management control : Refers specifically to the functional divisions within 
OSFA and the student aid programs and the types of quality control 
standards, information, reporting, and analysis required to improve 
management of the Pell program. 

• Operational control : Refers specifically to operational control of the 
student aid programs (particularly Pell Grants) and the types of informa- 
tion reporting, monitoring, and analysis required to improve the actual 
process of program delivery. 

The three-tiered analytic framework provides an effective lens through which to 

view the QC needs and information resources for the Pell Grant Program. The types 

of QC needs identified in the functional analysis clearly vary by level of organization 

for OSFA student aid programs. The functional analysis identified QC information 

needs at all three levels: 

• At the policy level, QC concerns relate primarily to the quality of overall 
program delivery. Outside of OSFA, in the Department of Education (ED) 
and the Office of Management and Budget (OMB), there is a concern that 
payment error in the Pell program be assessed on an annual basis. These 
same concerns are apparent within OSFA, along with an accurate aware- 
ness that critical parts of the policy cycle (planning, budgeting, and 
regulation) and program delivery (application through reconciliation) be 
performed on a timely basis. 

• At the management level , quality control for the Pell program is hindered 
by the lack of program emphasis in the formal organization structure of 
OSFA. The functional organization structure of OSFA makes program- 
matic developments more difficult because responsibilities for the program 
are spread across divisions with no formal linkages between critical 
program functions. The goals and objectives system currently being 
implemented by the Division of Systems Design and Development has the 
potential of overcoming this key problem with program management if 
appropriate QC enhancements are developed. 

• At the operations level , QC issues are more abundant. Branches and 
divisions have developed their own QC procedures, as needed, usually on an 
ad hoc basis. As a result, there is a wide variation in the awareness of 
staff within branches about QC issues. Some branches have developed 
fairly sophisticated approaches to QC, while others are confronted by basic 
problems. 

Given existing QC requirements and information resources it is possible to 
propose a preliminary design that identifies modular QC components that address 
system needs at each of these levels. Accordingly, a five-component QC system was 
proposed. This included: 

iv 



• A policy-level QC management information system to provides 

Analysis of QC corrective action options 

Analysis of problems with the program delivery cycle 

Analysis of problems with the planning, budget, and evaluation cycle 

• An external QC subsystem for annual assessment of overall payment error 
in the Pell program 

• An OSFA goals and objectives QC subsystem as a source of critical paths 
flow models for the policy cycle (planning, budget, and evaluation) and 
program delivery cycle for the Pell program 

• A student/disbursement QC subsystem for identifying error-prone insti- 
tutions and student profiles based on analysis of application and disburse- 
ment data 

• An institution review QC subsystem for developing QC standards and 
measures for postsecondary institutions 

• A processor /vendor subsystem for monitoring performance of the Pell 
processing contractor 

It is possible to integrate other student aid programs into this basic framework. 

Three of the subsystems—policy level QCMIS, OSFA QC, and institution review—can 

be modified to include both GSL and Campus-Based programs. The external QC also 

can be modified to include Campus-Based programs. Also, additional subsystems can 

be identified for the disbursement function in the other programs; 

• An interest payment subsystem for GSL 

• An FISAP QC subsv«tem for the Campus-Based fund allocation 

Based on the functional analysis and preliminary QC system design, strategies 
are considered for continuing the QC system design and testing task. First, it is 
essential that a sound systems methodology be used to develop QC subsystems (or 
modules) once they are selected by ED. Such a methodology is proposed in the final 
chapter. Second, a corrective actions component of the overall QC system must also 
be developed. A strategy for achieving this was also proposed. The next step in the 
QC system development process is the selection of QC subsystems for development. A 
working paper on this topic has already been prepared and submitted. 



CHAPTER 1 
INTRODUCTION 



Quality Control (QC) systems are essential to the sound management of Federal 
student assistance programs* This is especially true for the Pell Grant Program, the 
second largest student aid program administered by the Office of Student Financial 
Assistance (OSFA) and one prone to error and abuse* The annual overpayment in the 
Pell program, due to various institutional, student, and processor errors, is estimated 
to be approximately $300 million. The development of a QC system for the Pell 
program is complicated by these factors: 

• The delivery system is subject to change as a result of administrative or 
legislative action. 

• The current delivery system is already in operation and involves an 
extremely large number of primary actors. 

• Many of the potential avenues for corrective action in the current system 
only recently have been identified as a result of the Stage One QC study. 

In spite of these factors, the time is now critical for the development of a QC 
system for the Pell program, as it is for other student aid programs. The overall 
purpose of QC procedures and systems is to reduce error in the program through 
systematic development and refinement of administrative procedures and automated 
systems that can monitor error-prone functions and generate timely and policy- 
relevant analysis of these functions. This report represents significant progress 
toward this end. The purposes of this report are to; 

• Develop a conceptual framework for QC system development for the Pell 
Grant Program. 

• Provide a detailed functional analysis of the Pell Program and the status 
and the use of QC standards in the program. 
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• Provide a preliminary design for a modular QC system for the Pell 
program, including consideration of possible interfaces with other student 
aid programs. 

• To discuss strategies the Department of Education (ED) should consider for 
developing a QC system for the Pell Grant Program. 

Before detailing the contents of this report, it is important to define quality 

control as it has been used in this study. Specifically, the applicability of QC concepts 

to student aid is critical, since QC standards are not easily transferred from one 

sector to another. 

DEFINITION OF QUALITY CONTROL 

Quality control, while essential to almost any enterprise, is ambiguous and 
difficult to define precisely because as a concept it is applicable to such a wide range 
of activities. The essential element of a QC system, in either an industrial or social 
service setting, is the capability for detecting, preventing, and correc ting errors or 
any tendency toward errors that occurs in an operational setting. Implicit in this 
definition is recognition that: 

• It is essential to establish acceptable standards for the output (or products) 
of a system. 

• Standards must incorporate technical specifications prescribed by the 
provider of the service (or product) and expected by the recipient (or 
consumer). 

• A system for monitoring standards and correcting errors is necessary for 
the implementation of an effective QC system. 

• The purpose of a QC system is to increase the efficiency and effectiveness 
of the delivery system by reducing costly errors and increasing the quality 
of services (or products). 

During Stage One of the QC study, Advanced Technology examined the applicability of 
QC concepts and procedures to student financial assistance programs in order to 
develop a QC concept for the Pell program. The project team: 
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• Reviewed relevant literature on industrial and governmental QC systems 
and practices. 

• Interviewed QC personnel from five social service agencies operating at 
the Federal, state, and local levels. 

• Conducted a workshop/discussion with these personnel and ED staff. 

PURPOSES OF THE STUDENT AID QC SYSTEM 

From Stage One activities, a working concept of QC was developed for student 
financial assistance programs, particularly the Pell Grant Program. It is essential that 
a QC system for student assistance programs enable ED to: 

• Develop standards and measures for monitoring the delivery of student aid. 

• Measure performance of student aid delivery against specified measures. 

• Determine and monitor errors in eligibility determination and award 
processing* 

• Identify sources and probable cause of errors to plan corrective actions- 

• Develop corrective action procedures as an integral part of the processing 
functions. 

• Develop standards and measures for monitoring the results of corrective 
actions. 

• Ensure that various actors (e.g., processors or institutions) are operating in 
accordance with specified procedures, regulations, and standards. 

• Report appropriate QC information to ED personnel on a timely basis. 

To accomplish these functions, ED will need to develop QC procedures 
performed on an ongoing basis as the award process is taking place (front end) and on 
an audit basis after the award process has been completed (back end). For the Pell 
Grant Program, it is desirable that front-end procedures be developed to reduce the 
amount of error prior to the disbursement of funds. It is also necessary to have a 
back-end component to ensure that these corrections have actually taken place, as 
well as to perform analysis and develop corrective action for future years. The 
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certainty that corrective action will be implemented and monitored is an essential 
component of the QC system. 

The QC cycle, illustrated in Figure l-l, identifies necessary steps in the QC 
corrective action process* The first five steps in the cycle require a thorough analysis 
of the delivery system and supporting information systems for student aid programs. 
The final four steps in the cycle require an understanding and a thorough analysis of 
tb? management and policy processes in OSFA, in order to develop decision 
mechanisms that will result in corrective actions in critical areas* This paper 
primarily addresses the issue of designing a QC system that will: 

# Define the sampled subject for control. 

# Define a unit of measure. 

# Establish standard of performance. 

# Create a measuring device or procedure. 

The subsequent steps in the QC cycle will be addressed as the project moves into 
the development of QC modules. This will include consideration of n igement 
procedures and policy coordination mechanisms that will result in completion of the 
final steps for selected targets of opportunity. However, there is some preliminary 
discussion of the requirements of a corrective action system in the final chapter. A 
formal process for initializing and implementing corrective actions will be required to 
complete the QC loop in the overall system design* 

CONCEPTUAL MODEL FOR THE QC SYSTEM 

Although a detailed conceptualization of the Pell QC system is an objective of 
the Stage Two study, it is first necessary to conceptualize the basic components of 
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such a system. Conceptually, a QC system for the Pell program would combine a core 
QC Management Information System (QCM1S) for monitoring and tracking the award 
process with a series of subsystems for the major actors in the award process; the 
application processors (Central and MDE), OSFA (Central and Regional), institutions, 
and students* The subsystems would both monitor performance against standards and 
determine error levels within each subsystem* Such a system could also have an 
external QC subsystem, which would provide an external check on the program by 
sampling an appropriate group of recipients to determine if they received the correct 
award amounts* A simplified conceptual model is illustrated in Figure 1-2* 
Steps in Building a QC System 

In practice there could be more than these four subsystems. Each subsystem 
must interface with both manual and automated systems and, therefore, must combine 
elements of both. The development of a QC system for the Pell program will be a 
complex process requiring: 

• Detailed analysis of the Pell delivery system 

• Development of a conceptual model for an overall QC system 

• Systematic development of QC subsystems that integrate into the delivery 
system 

• Development of a central monitoring and tracking system, the QCMI5 
THE REPORT 

This report represents completion of the first step in the development of a QC 
system for the Pell Grant Program* The report provides a detailed functional analysis 
of current QC procedures in OSFA student aid programs as well as conceptualization 
of a framework for the development of the QC system. 
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The report has five chapters. After the introduction, Chapter 2 presents the QC 
system development framework used in the Stage Two study. It explains the strategic 
modular approach that will be used to design the QC system for the Pell program. 
Chapter 3 provides a functional analysis of the QC requirements for a Pell Grant QC 
system. It analyzes QC requirements at the policy, management, and operations levels 
for the Pell Grant Program. Chapter * presents a preliminary modular design for a 
Pell Grant QC system. This system is composed of a series of modular components 
that can be developed independently, in an incremental phased fashion. Chapter * also 
considers how the QC system design can be expanded to include other OSFA student 
aid programs. Finally, Chapter 5 considers general strategies for proceeding with the 
system design effort, including preliminary consideration of the policy mechanisms 
required to cevelop corrective actions designed to reduce error in the delivery system. 




CHAPTER 2 
QC SYSTEM DEVELOPMENT FRAMEWORK 

The development of QC procedures and systems in environments that already 
have operating systems, in government or industrial settings, is a complex process. It 
requires an understanding of organizational dynamics as well as systems development 
methodologies. In the case of a Federal student aid program, QC system development 
also requires an understanding of the policy process, since a QC system must provide 
policy-relevant information. In the present context the selected QC system develop- 
ment strategy must also accommodate policy decisions that might change the basic 
structure of student aid delivery. As a result, it is essential that a QC system 
development framework be used that provides sound methodology and conceptual 
design, as will as flexibility with respect to a changing organizational setting. 

This chapter presents the basic framework that will be used to develop and test 
the QC system for the Pell program. Four topics are considered: 

• Generic Approaches: A review and evaluation of approaches to QC system 
development, including selection of a general methodology. 

• System Design; Preliminary assessment of alternative Pell Grant QC 
system configuration options 

• Organizational/Information System Perspective: An information system 
pe-spective of the organizational environment in the Office of the Student 
Financial Aid, considered as a basis for the functional analysis and system 
design 

• Analytic Framework: Based on these three considerations, an analytic 
framework is specified for the QC system development task. 

GENERIC APPROACHES TOQC SYSTEM DEVELOPMENT 

To set the stage for ED decisions about the design and development of the Pell 
Grant QC system, it is important to consider three generic approaches to QC system 

2-1 
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development. One approach represents the status quo for student aid programs. The 
other two approaches could be used to develop a new QC system for the Pell program 
or other student aid programs. The first is characterized as the incremental bottom- 
up approach, the second is the comprehensive approach, and the third is the strategic 
or modular approach. A discussion and comparison of these generic approaches 
follows. 

Incremental Approach 

The incremental, bottom-up approach to QC systems development typifies the 
approach used in most government agencies, educational institutions, and industrial 
settings. The incremental approach assumes that functional subunits within an 
organization have responsibility for corrective actions and procedures in their areas of 
responsibility. Corrective action and QC system development are usually truly 
incremental in this context, and, therefore, seldom the outcome of coherent planning. 
It is possible, however, to specify the steps typically followed if the incremental 
approach is taken: 

• Functional subunits (division, branch, etc.) discover error -prone areas 
through problems with the system in operation. 

• Plans for corrective actions (new procedures or system changes) are 
developed and based on needs as they arise; usually a limited range of 
options are considered. 

• Managers seek new resources for corrective actions— system development 
or implementation of new procedures— but problems that involve an inter- 
face with other subunits (divisions and branches) are often ignored unless 
the resource issue can be resolved. 

• QC procedures are developed and implemented as time and resources 
permit. 

For the Pell Grant Program, the incremental approach typifies the approach used 
to date in the area of quality control. While the Division of Quality Assurance (DQA) 
has implemented some QC reporting requirements in the past, there have been no 

2-2 
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systematic attempts to devlop a QC system for Pell or the other student aid programs. 
The types of problems likely to emerge if the incremental, bottom-up approach is used 
are: 

• Production of tasks in the necessary time frame taking priority over 
smooth operation and reduction of error in the system and efficient use of 
personnel and other resources 

• ^consistency and va* a. ion in the ways functional units deal with quality 
control problems 

• Inadequate development of quality assurance procedures in functional units 

• Acute problems in the quality control of products that cut across 
functional subunits (or that involve more than one division/branch) 

When acute problems emerge in an operating system, which the Stage One QC 
study indicates is the case for the Pell program, it becomes necessary to take a more 
systematic approach to QC system development. When major problems emerge, 
incremental changes (with bottom-up control) are no longer adequate. 
Comprehensive Approach 

The comprehensive approach assumes that anything that can go wrong with the 
operating system will go wrong; therefore, it is important to identify every possible 
error in the system and design corrective procedures. The basic steps involved in this 
approach are outlined as follows (with the application of each step to the Pell Grant 
Program considered in parentheses). 

• Identify the major subsystems of the programs (for Pell Grants this would 
include eligibility determination, award processing, fund disbursements, 
and so forth). 

• Identify the major actors or components for each subsystem. (For the 
eligibility subsystem this would include students/parents, institutions, and 
application processors.) 
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• For each actor in each subsystem, identify acceptable measures of possible 
errors in the system. (For students and parents in the eligibility system 
subsystem this will include identification of standards for application 
errors.) 

• Define measures for each set of standards. (This would include identifica- 
tion of data elements and procedures for information collection.) 

• Determine the components of each subsystem that merit development and 
inclusion in the QC system. (Evaluate the feasibility of including the 
various information sources in a QC data base.) 

• Proceed with system development on the select subsystems. (Design and 
develop procedures for implementation of selected components of each 
subsystem.) 

For the Pell Grant Program, the comprehensive approach would result in an 
elaborate QC system that imposes new data collection procedures on top of the 
already existing system. It would permit the development of a separate or stand-alone 
QCMIS. It could also be used to produce QC manuals for training ED personnel 
(Central and Regional) and institutional representatives in the practice of QC in the 
Pell Grant and other student aid programs. To the extent that the system used 
automated data collection and analysis procedures, it would be labor intensive since 
virtually an entire new set of procedures would be needed for each component of each 
subsystem; and sophisticated data base management procedures may be needed 
centrally, depending on how much of the system is eventually implemented. 
Strategic Approach 

This approach assumes that the major sources of error in the Pell Grant Program 
can be identified and that corrective action should be made in these areas through the 
development of modular components of an overall QC system. The basic steps 
involved in the strategic approach are outlined as follows (with applications to the Pell 
Grant Program): 

• Identify significant sources of e. ror in the program. (For the Pell Grant 
Program this was accomplished in the Stage One QC study.) 
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• Conduct a functional analysis of the operating system, including informa- 
tion requirements, linkage structures, and breakdown points. (Identify the 
places in the system where corrective action can be taken and monitor 
progress.) 

• Conceptualize a QC system with modular components designed to detect 
and monitor error-prone functions. (Conceptual system design would 
consider modular subsystems that Interface the QCMIS with subsystems of 
the Pell Grant system, e.g., Processor, funds disbursement.) 

• Select and prioritize modular subsystems for development. (Identify time 
frame for specifications, design, development, and installation of each 
selected subsystem.) 

• Proceed with systems development for selected subsystems (e.g., develop 
procedures and systems manuals, user manuals, system specifications, and 
software specifications, as necessary). 

• Perform system tests on modular subsystems as they are completed. (This 
would be determined as the work schedule was developed and approved.) 

For the Pell Grant Program, the strategic approach to QC development would 

permit the incremental implementation and testing of modular QC subsystems 

designed specifically to reduce errors in the system. The modular approach could also 

permit the use of up-to-date electronic technology utilizing preexisting data sources, 

where appropriate, rather than developing new data sources and reporting procedures. 

In other instances it might result in more systematic analysis and reporting on data 

currently collected. The principal advantage of the modular design is the use of 

preexisting data sources, where appropriate, with enhanced reporting and monitoring 

capabilities. However, in some areas new data sources would be needed. 

Evaluation 

For the purposes of this project, both the comprehensive and strategic 
approaches to QC system are viable approaches. This section compares the compre- 
hensive and strategic approaches to QC system development using the following 
objectives of the proposed QC system as evaluative criteria. 
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• Reduce Error in the Pell Grant Program . 

• Maintain Flexibility to Add Additional Programs. 

• Pilot Test QC System Components. 

• Produce Sound QC Methodology. 

The two approaches vary in their potential for reducing errors, particularly the 
overpayments identified in Stage One of the QC study. The comprehensive approach 
would build in an elaborate procedure for identifying all possible errors but would also 
introduce complicating factors such as possible ove* emphasis on less important issues 
and time delays in monitoring key factors due to the implementation and operation of 
new data collection procedures. This would severely constrain the implementation of 
front-end corrective actions, since the excessive staff commitment to system 
implementation would limit staff resources available for analysis of corrective action 
options. The strategic approach would be targeted at critical problems in the design 
stage; it would also attempt to use, wherever possible, preexisting data sources. This 
would ensure that systems development work would be targeted at the most critical 
timely information. This approach could also be oriented toward front-end and back- 
end correction. Therefore, the strategic approach would maximize the potential for 
reducing program costs in a shorter time frame. 

The two approaches also vary considerably in their flexibility to add additional 
programs to the QC system. With the comprehensive system it is possible to repeat 
the same procedure for other student aid programs or to modify the system and data 
correction procedures to include other student aid programs. In contrast with the 
strategic approach ED can create new modular subsystems for other student aid 
programs, as part of an overall integrated QC system., In fact, the development of 
modular subsystems could be staged to coincide with changes in the delivery system. 
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The strategic approach would also provide ED with the opportunity to Pilot test 
QC system components at an earlier stage in the process. Inherent in the compre- 
hensive approach is the design of new data collection procedures that involves defining 
new data elements and devising new collection procedures, which take time to 
implement and test. The strategic approach emphasizes the use of existing data 
sources, a time-saving procedure, and the phased development of QC subsystems- Ttis 
should allow for pilot testing components of the QC system earlier in the development 
process. 

Both approaches would produce sound QC methodologies but would have 
different results with respect to the type of methodology developed and used. The 
comprehensive approach would result in the same methodology being applied to each 
subsystem of the Pell program, with critical subsystem components receiving the 
appropriate levels of attention. Since it is an elaborate system review process, a 
well-defined procedure would be necessary, which may also apply to other student aid 
programs. The strategic approach would have to use sound systems development 
procedures in the conceptual development of the QC system and modular subsystems. 
As a part of the overall functional analysis, the contractor would evaluate the value of 
the QC procedures already in use, for example, in manuals currently distributed to 
institutions, before developing new procedures. Therefore, the procedures developed 
as a result of the project would be specifically oriented toward the existing system, 
but the procedures used for the entire project would be applicable to all Title IV 
programs. 

In summary, given the objectives of the QC system design and testing phase of 
the Stage Two QC study of the Pell Grant Program, the strategic approach to systems 
development increases the prospect of: 
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• Reducing errors in the Pell program 

• Adding other student aid programs to the QC system 

• Pilot testing QC components earlier in the study 

• Developing a sound generalized methodology 

At a recent meeting on the QC system design, DQA officials expressed a desire to 
achieve these ends. Advanced Technology concurs with this emphasis and recommends 
that the strategic approach be used in the Stage Two study. 

QC SYSTEM DESIGN 

It is not possible at this stage in the study to provide a detailed description for 
conceptualization of the QC system. Nor is this desirable, since ED still must make 
some basic decisions about the type of QC system desired, and the functional analysis 
included in this report will be a major input to the system design. This section 
considers alternate configurations for the QC system relative to the degree of 
automation and use of preexisting information. Before considering these options, it is 
necessary to consider the purposes of the QC system. 
QC System Purposes 

A basic conceptual model of the Pell Grant Program QC system was discussed 
briefly in Chapter 1. This model is reconsidered in Figure 2-1. Each of the four major 
subsystems has two purposes: 

• Each provides information on the functioning of the subsystem. 

• Each is the object of quality control. 

Ideally, information would flow back from the QCMIS about the types of 
corrective action that should be taken. This conceptualization is oversimplified since 
it ignores the information for each of the subsystems. The source of information and 
whether it will come from existing and new sources are still to be determined. It is 
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CONCEPTUAL MODEL OF THE PELL GRANT QC SYSTEM RECONSIDERED 
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possible, however, to describe an idealized form of the functions of each of the 



subsystems, and the QCMI5. 

• QCMISt This central QC information system would receive routine 
summary information uoir, «ach of the subsystems. Reports would be 
analyzed, possibly th ough ,bsystem simulation of the program delivery 
process, to identify prcbierr ^ in the overall delivery system (either as early 
warnings or follow-up reports) and alternative corrective actions that can 
be taken (front end or back end). This subsystem would also be used to 
monitor the impact of corrective action. 

• Student Subsystems! Would combine and analyze information about 
program applicantsfmonitoring and evaluation reports would be generated 
about student error and used as a basis of policy decisions about corrective 
actions; information about corrective action procedures would be fed back 
into the subsystem. 

• Institutional Subsystem! Would provide a mechanism for monitoring QC 
measures and standards for institutions' reports from the institutional 
review function and would provide reports on trends to the central QCMIS; 
would also integrate information about corrective actions taken. 

• Processor Subsystems: Would develop a series of QC subsystem checks to 
provide input to the QCMIS for policy decisions about the program and 
would alert policymakers to potential problems with the processing 
function. 

• OSFA Subsystem: Would provide a routine procedure for monitoring OSFA 
management of the Pell program; would identify points where corrective 
actions are needed in the system. 

• External QC Subsystem: Would provide an external quality control check 
on the program by sampling the program's recipients for purposes of 
determining whether awards are correct and, if not, what the sources of 
error are in the program. This component would provide feedback to the 
other major subsystems through the QCMIS. This could be a periodic 
replication of the Stage One QC study. 

These descriptions are an idealized and simplified version of how a QC system 

might work . In reality, the QC system will have to utilize information from a wide 

range of sources, including thousands of institutions and millions of students. It is 

highly likely that QC relevant information can be generated from existing data 

sources. However, in its conceptualized form, a QC system for the Petl program will 
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provide a mechanism for monitoring and correcting error and tendency toward error in 
each of the major subsystems * 

• Students/applicants 

• Institutions 

• Application processor 

• OSFA 

Alternative QC System Configurations 

While it is neither possible nor desirable at this early stage to provide a detailed 
description of what the QC system will actually include, since this is an objective of 
the QC system design and testing project now underway, it is important to speculate 
about what the QC system might look like and how it will interface with the Pell 
delivery system. It is particularly difficult, without undertaking the functional 
analysis of the Pell delivery system, to specify the extent to which the QCMIS will be 
a manual or automated system and the extent to which it will use new or existing data 
sources. Two issues help distinguish between optional configurations for the QC 
system: 

• The extent to which the new system is manual or automated 

• The use of new or existing data sources 

These issues can be used to generate four discrete configuration options. The criteria 
used to evaluate these options are: 

• Feasibility of the system design (Can it be done?) 

• Potential for reducing error (Will it save money?) 

• Developmental costs (How much?) 

• Interface with the delivery system (Will it work?) 
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At this point, consideration of the actual configuration of the QC system is 
preliminary. However, it is possible to outline and compare a few options, using these 
as evaluative criteria. As a part of the system development project, a rigorous and 
detailed analysis is necessary. This preliminary analysis helps explain the emprasis of 
the current QC system design effort. 
Option 1; Manual Enhancements 

It is possible to develop a QC system that essentially provides manual enhance- 
ments to the existing Pell delivery system. In this case, the system development 
activity would include detailed specifications of reporting requirements for each of 
the major actors in the Pell Grant Program. The reports would be entered into a filing 
system in the Division of Quality Assurance ami used as a means of monitoring and 
tracking progress on certain key areas identified during the system development 
process. Either the comprehensive or the strategic approach to the system develop- 
ment just outlined could be used to develop the manual enhancements, although the 
comprehensive approach is easiest to adapt to this option. The major problems with 
this option are that it would add to the reporting burden for the major actors and that 
excessive time delays would be built into the system. 

Option 2: Combined Manual/Automated QC System Using New Data Sources 

This option would essentially take the above-stated concept an additional step- 
Where appropriate data collected through the QC process could be automated, some 
files would be accessible for automated analysis and reporting* Others, particularly 
periodic summary reports using aggregated data, would remain manual* This option is 
an improvement on Option 1, using automated technology as appropriate within th* 
framework of creating a new data collection system. It has the potential of providing 
data on a somewhat more timely basis, but it would provide an additional layer of 
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reporting on top of the existing delivery system. It would use either system 
development approach but would be more adaptable to the comprehensive approach. 
Option 3; Combined Manual /Automated PC System UsinR Existing Data Sources 

This option would have some of the same features as Option 2 but would 
emphasize new analyses of existing data sources rather than development of entirely 
new data collection procedures. This would involve utilizing, where appropriate, data 
the major actors currently generate or that could be generated relatively easily 
through modification of current practice. In some instances this might involve 
reporting progress in corrective actions and in the sreed of handling certain types of 
problems. For example, if the IRS Form 1040 cross-check were built into the Pell 
processor contract, a system for monitoring and reporting on this activity could be 
built in as well. This approach would be flexible enough to add other student 
assistance programs as necessary. In this w;.y a QCMIS subsystem could be 
constructed that dealt with critical points in the delivery system. This option would 
be most adaptable to the strategic approach for QC system development. 
Option »; Fully Automated Integrated QC System Using Existing Data Sources 

This option would take the approach just identified to the fullest possible degree 
of system automation. Such an option would integrate the core QCMIS with systems 
that interface with the major actors' operating systems. Monitoring, comparing 
results to specific standards, taking routine corrective actions, and reporting could be 
built into the system. Such an approach would be dependent on automation of most 
components of the Pell delivery system. Other student aid programs could be 
integrated into the system, depending on whether and how the student assistance 
delivery system was eventually redesigned. This option could use a variation of either 
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the strategic or the comprehensive approach to systems development but would have 
to be done in combination with the redesign of the entire delivery system. 
Assessment 

A functional analysis of the Pell Grant system is required to fully evaluate these 
options* In the absence of such an analysis, only preliminary judgments are considered. 
Once the functional analysis of the program is complete, it will be possible to develop 
these and/or other options in greater detail, in which case it will be possible to provide 
a detailed evaluation. Figure 2-2 summarizes the preliminary assessment of the four 
generalized options. This assessment uses the evaluation criteria just identified. 

On the basis of the preliminary assessment it is possible to make an initial 
judgment about which QC system option is most desirable. Option 1 would probably 
have relatively modest results, with respect to its potential for reducing error, and 
moderate developmental costs, and it would not be integrated with the delivery 
system* Option 2 would increase the potential for reducing error and the develop- 
mental costs. Option 3 has high potential for reducing error, would require moderate 
developmental costs compared to Options 2 and *, and would be partially integrated 
into the delivery system. Option 4, while having the highest potential for reducing 
error, does not appear feasible at the present time. As the Stage Two QC study 
progresses, however, it could be further evaluated as part of the delivery system 
redesign. Of the four options, Option 3, a combined automated/manual system using 
existing data sources, appears to be the most desirable QC delivery system configura- 
tion. 

This preliminary assessment indicates that Option 3 is the desirable configura- 
tion for the Peil Grant QC system; a full evaluation of the precise configuration will 
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OPTION 1 



OPTION 2 



OPTION 3 



OPTION 4 



Oescription 
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Feasibility 



Potential for 
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Manual QC System with 
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Moderate (Can be 
implemented with 
existing delivery 
system— manual 
analysis may take too 
long to be useful^ 

Low (Delays built into 
reporting limit moni- 
toring front-end cor- 
rective action* Imple- 
mentation may not lead to 
error reduction.) 



Developmental Moderate (Requires 
Costs developing entirely 

new system.) 



Interface with 

Delivery 

System 



Not Integrated (Result 
is creation of new, 
parallel system.) 



Combined Automated/ 
Manual QC System with 
New Data Sources 
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PRELIMINARY ASSESSMENT OF QC SYSTEM OPT!ONS 
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be necessary at a latter point. However, it appears that a combined manual/auto- 
mated QC system using existing data sources is desirable. This also fits best with the 
needs of the Office of Student Financial Assistance. This is considered in more detail 
in subsequent chapters- 
ORGANIZATIONAL FRAMEWORK 

The development of a QC system for the Pell Grant Program, or for other 
Federal student aid programs, will take place in a complex organizational setting. 
Whether the actual delivery system is stable or changes as a result of new legislative 
actions, the QC system must interface with the bureaucratic organization in the 
Office of Student Financial Assistance. To adequately consider the information needs 
of a complex organizational setting, it is necessary to consider the different levels of 
decision in an organization and how they interrelate. Information system planners 
generally distinguish between the levels of the decision-making hierarchy (Anthony, 
1965; Bassett, 1979; Blumenthai, 1969; and Weathersby, 1975). 

• Strategic Planning (and Policy) is the process of deciding on objectives 
which can be expressed as programs in large organizations, changes in 
objectives, resources to be used, and policies that govern the acquisition, 
use, and distribution of resources. 

• Management control is the process by which managers assure that 
resources are used effectively and efficiently in the accomplishment of the 
organization objectives. 

• Operational control is the process of assuring that specific tasks are 
carried out efficiently and effectively. 

Each level of decision making has different information needs. For large student 
aid programs, such as the Pell program, there are many actors involved in the decision 
process, ranging from the President and Congress, at the strategic level, to insti- 
tutional and operational units in OSFA at the operational control level. This 
information system perspective on or c anizational decision making can be applied to 
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OSFA and the Pell program. Figure 2-3 depicts an information system perspective of 
OSFA student aid programs using this three-tiered organizational framework. The 
level of organization^ volved in the decision making about student aid programs 
ranges from the President and Congress, who make major policy decisions, to the staff 
in OSFA and higher education institutions actually responsible for operation of the 
programs. As recipients, students are not directly involved in the management or 
control of the program. Some actors are involved in two levels of decision making. 
For example, the Deputy Assistant Secretary for Student Financial Aid is involved at 
both the strategic and management levels, while other OSFA administrators are 
involved at both management and operational levels. 

The information system perspective on the OSFA hierarchy provides a 
systematic way of viewing the types of decision objectives that are important at each 
level, as well as the types of decision inputs, information sources, and decision jutputs 
that are considered important at each level. Clearly, each level of the orgaitization is 
dependent on a flow of information and analysis. The problem is that types of 
information required for each level of decision making vary; therefore, the degree of 
specificity in reporting and timeliness of reports required at each level will also vary. 
At the operational levels, managers and personnel are concerned about the basic types 
of data elements (application forms, IRS forms, etc.), as well as the timeliness of 
processing to ensure that students receive awards. At the management level, 
summary reports and detailed analysis of patterns and trends are required to provide 
management with appropriate information for the assessment of operating objectives. 
At the strategic level, special one-time reports are needed to determine if policy 
objectives are being met or need to be changed; simulations can be used to forecast 
the consequences of alternative policies; and unrestricted inquiries may be necessary 
when major problems or new initiatives arise. 
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This perspective can also be used to illustrate how a quality control system 
would most effectively interface with the organization hierarchy in the Office of 
Student Financial Assistance* Figure 2-4 presents an organization perspective on how 
the QC system would interface with the OSFA hierarchy* The external QC system 
could provide a formal mechanism for conducting inquiries into the way the system is 
working. The QCMIS would combine routine reporting from the subsystems with a 
simulation capacity that would use QC data to analyze the impact of policy issues. 

The perspective provided by the organizations and information systems frame- 
work is highly adaptive to the issues already considered: 

• The use of a strategic, modular approach to QC system development 

• The development of a combined automated manual system that uses 
existing data to the extent possible. 

In fact, the organizational framework provides insights into how the data 

gathered in the functional analysis should be analyzed to meet the needs of the 

Department of Education, The analysis and the organizational resources, management 

objectives, and QC standards of the Office of Student Financial Assistance must 

reconcile the fact that information resources and needs vary considerably by level of 

organization. 

ANALYTIC FRAMEWORK 

The analytic framework used in this report is derived from a synthesis of three 
iss-jes. First, a strategic approach to QC system design and testing is the most 
desirable approach for this activity. The basic steps implied in the strategic approach 
are: 

# Identify major sources of error. 

# Conduct a functional analysis of the current delivery system. 
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AN ORGANIZATIONAL PERSPECTIVE OF THE PELL QC SYSTEM 
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• Conceptualize QC system modular components. 

• Select and prioritize modular components. 

• Proceed with development of selected components. 

• Perform system tests on modular components as they are developed. 

This report represents the second and third steps of this process. The functional 
analysis of the Pell Grant delivery system is included in Chapter 2, while the 
preliminary conceptualization of a QC system for the Pell program is contained in 
Chapter 

Second, during the data collection stage of the functional analysis of the Pell 
delivery, emphasis was placed on documenting and analyzing current sources of 
information and management procedures, since the most viable configuration option 
appears to be a combined automated /manual system that uses existing data* This 
would increase the possibility that the new system would be resource saving, using and 
enhancing existing data and management practices, rather than resource draining , 
placing new demands on the current delivery system. The basic data gathering 
techniques used during the functional analysis have emphasized using existing 
resources; 

• Interviews emphasized documenting existing procedures, processes, and 
sources of information. 

• Reviews of ED/OSFA documents and program regulations* 

Third, analysis of the information collected will use the organizational per- 
spective on information systems. Three levels of information sources, resources, and 
requirements are analyzed in subsequent chapters: 

• Strategic policy : Refers to the policy and political arena and the types of 
quality control information, reporting, monitoring, and analysis required 
for improved policymaking. 



2-21 

39 



• Management control! Refers specifically to the functional divisions within 
OSFA and the student aid programs and the types of quality control 
standards, information, reporting, and analysis required to improve 
management of the Pell program. 

• Operational control? Refers specifically to operational control of the 
student aid programs (particularly Pell Grants) and the types of informa- 
tion reporting, monitoring, and analysis required to improve actual process 
of program delivery. 

The subsequent chapters, which provide a functional analysis and preliminary 

system design for the Pell program, will use this three-tiered perspective. The 

development of an effective QC system that uses existing resources, and that results 

in resource savings, is dependent on a thorough understanding of the QC requirements 

and needs at each of these levels in the OSFA organization. 
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CHAPTER 3 
FUNCTIONAL ANALYSIS: 
PELL PROGRAM DELIVERY SYSTEM 



QC system development in the current Pell program delivery system requires a 
thorough functional analysis of the Pell delivery system within the Office of Student 
Financial Assistance. This chapter undertakes such an analysis for OSFA Central and 
Regional Offices. The other major actors— institutions and application processors- 
have not been included in this analysis. However, they will be analyzed as the design 
project progresses. The purpose of this analysis is to identify and assess the QC 
information requirements for the Pell program. This chapter analyzes the Pell 
delivery system, including information systems and needs for improved QC procedures 
and systems, from three perspectives. 

First, the policy context is considered. Many of the basic decisions about the 
Pell Grant Program are made in the political arena, as a result of analysis by ED, 
Office of Management and Budget (OMB), and congressional staff. Usually this takes 
place either through the annual budget process or through the reauthorization of the 
Higher Education Act, the legislation which sets the basic parameters for Pell and 
other student aid programs under Title IV. This analysis considers the types of QC 
information required in the policy arena. 

Second, the management of student aid programs in OSFA is considered, 
including the types of QC information procedures required for effective management 
of the Pell program. This includes analysis of the organization of the Office of 
Student Financial Assistance (OSFA) and the management procedur r tly being 
implemented in the office. 
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Finally, the actual delivery system for the Pell program is considered. This 
analysis considers: 

• The function of major organizational units involved directly in the Pell 
program delivery system 

• The quality control procedures currently in place 

• The information resources within the wit 

• The information needs of the unit 

THE POLICY CONTEXT 

The QC issues for the Pell program that concern Federal policymakers are a 
direct outgrowth of the program's history. The °ell Grant Program was created as the 
Basic Educational Opportunity Grant (BEOG) Program in the Education Amendments 
of 1972, which reauthorized the Higher Education Act of 1965. The purpose of the 
BEOG and other student aid programs authorized under Title IV was to promote equal 
educational opportunity. The funding of the program began in FY73 with the freshman 
class of 1973, At first the delivery system was entirely a paper process, with students 
making applications directly to the application processor, which was the American 
College Testing (ACT) Service. Each subsequent year until FY76 an additional 
entering class was added to the program. The award levels for the program (the 
maximum award allowed per student) were also increased. As a result, the costs of 
the program grew dramatically, from $69 million in FY73 to $959 million in FY76. 

The accuracy of the projected cost of the program is perhaps the major QC issue 
that concerned congressional and administration policymakers during the early years 
of the program. As a result, there was a rapid proliferation of micro-simulation 
models for the program. At first, cost projection models used for other student aid 
progjams were adapted to the BEOG program. But since Pell is basically an 
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entitlement program, these flow distribution models did not work. Pell is a demand- 
driven program, like other social programs and unlike other student aid programs. 
Within a year, an enrollment- based model was developed for the Pell program; it used 
the National Center for Education Statistics (NCES) enrollment projections for a basis 
of program cost projections. Different versions of this model were operated by HEW 
and other agencies, further confusing the political process. In the late 1970s, HEW 
started work on a new model that would use prior year application data. This became 
operational for FY80j a version of the model is now housed in OSFA, Two additional 
models were developed by the Congressional Budget Office and the College Entrance 
Examination Board. OSFA has also funded the development of a cost simulation model 
for all student aid programs. The combination of models has gradually increased the 
awareness of policymakers about the complexity of esimating costs for the program. 
The budget etimates for the program have always been off target due to a combination 
of policy and estimation problems. 

During the 1970s, the problems with estimating program costs preoccupied senior 
policymakers and analysts in HEW and Congress. The delivery system for the program 
also underwent a rapid evolution. Since congressional decisions about budgets and 
program regulations usually delayed starting the program each year, e.g., the 
preapplication process, the basic issue facing the Bureau of Student Financial 
Assistance (before ED was formed, OSFA was a bureau) was to deliver Pell Grants 
within a reasonable schedule. Quality control— identifying and reducing error in the 
program— was not a high priority. 

When the Department of Education was formed in 1979, the administration of 
education programs was modified significantly. While controversy continued over 
program budget estimates, the student aid delivery system became the subject of 
study. For example, the new Office of Management and Evaluation (OME) took a 
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direct interest in the quality of the student aid delivery system, and senior officials in 
OME circulated a paper prepared for the Secretary of Education, proposing that a 
major new system be developed for all student aid programs. Simultaneously, OSFA 
developed plans for the QC study of the Peii program. The initial findings of the Stage 
One study added to speculation that there were major QC problems and sources of 
significant error in the Pell delivery system. 

The Reagan Administration began to raise basic questions about the quality of 
entitlement programs during 198 1. Currently, there is a heightened concern about QC 
in the Pell delivery system in the policy arena. Specifically: 

• Officials in ED's Office of Planning Budget and Evaluation expressed 
concern that the QC system provide good analysis of delivery system 
options, such as the cost and benefits of validation by institutions or the 
processor. 

# All of the senior officials interviewed outside of OSFA expressed concern 
that significant changes were needed in the student aid delivery system and 
that development of a full-scale QC system in the current delivery system 
would not maximize the potential for error reduction. 

Currently, senior policymakers and analysts in Congress and the administration 
are dependent on QC information on student aid programs provided by OSFA. A 
senior-level Credit Management Task Force was recently formed in ED to plan for 
major changes in the entire student aid delivery system. The deliberations of this 
group, the reactions of the President and OMB officials to error in the program, the 
escalating costs of all entitlement programs, and the awareness that marginal 
improvements in QC can be implemented through the budget process are all factors 
influencing the policy-level requirements for QC information. 

During the recent round of interviews with ED and OMB officials about QC in 
Pell and other student aid programs, wide-ranging viewpoints were expressed about QC 
information needs in the policy arena. The two critical issues raised in these 
interviews were: 
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t OMB officials expressed concern that QC includes a system for annually 
measuring errors in the awards made in the Pell program* 

• OPE officially expressed a desire for timely analysis of corrective action 
options. 

While micro-simulation models for estimating program costs are operated 
outside of OSFA— e.g., OPBE and CBO— the collection and analysis of information on 
the quality of program delivery is an OSFA function. Therefore, a policy-level QC 
management information system that provided information and analysis directly to 
senior administration officials would fill a glaring gap that now exists in the current 
system for collecting and analyzing information on the quality of program delivery. 
Some of the policy relevant requirements for such a system would be to provide: 

• Annual assessment of the level of error in Pell award 

• Analysis of the quality of program delivery in OSFA 

• Timely analysis of policy-related issues for the budget and reauthorization 
processes 

THE MANAGEMENT ENVIRONMENT IN OSFA 

This section considers the management environment in OSFA and the QC 
resources and requirements considered important for the improved management of 
OSFA programs. Three topics are considered: 

• Organization of the management process in OSFA 

• Management processes and information resources 

• Quality control requirements 

All of these issues obviously interrelate. The discrete treatment of these three 
topics is necessary for a complete functional analysis of the OSFA environment. In 
this discussion district the Pell program is not treated separately. Since OSFA has a 
functional organization, the distinct analysis of the organization and management of a 
single program is complex. 
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OSFA Organization 

OSFA has a functional organization structure, with management responsibilities 
assigned according to function rather than program. A simplified organizational chart 
for OSFA is diagrammed in Figure 3-1. The Deputy Assistant Secretary for Student 
Financial Assistance heads OSFA* There are six divisions: 

• Division of Policy and Program Development (DPPD): Responsible for 
policy analysis, program planning, and regulation development for each of 
the programs* The Pell Grant Branch maintains the application processor 
contract for the Pell program, in addition to the responsibilities for policy 
and regulation* 

K 

• Division of Pr&gram Operations (DFO): Responsible for operations of Pell, 
Campus-Based, and G5L programs, the Pell Grant Branch is responsible 
for the regular alternate disbursement system. 

j — 

# Division of Certification and Program Review (DCPR) s Responsible for 
audit and review functions in OSFA. Sub units are not organized according 
to program. 

# Division of Systems Design and Development (DSSD) : Responsible for 
system design and development in OSFA. Currently, maintains an OSFA- 
wide goals and objectives systems. The division does not have a separate 
branch for the Pell program. 

• Division of Training and Dissemination (DTDh Responsible for training of 
OSFA staff and dissemination of program and consumer information to ED 
central and regional offices and to institutions. Not organized according to 
program. 

# Division of Quality Assurance (DQA) : Responsible for QC and quality 
assurance for student aid programs. Not organized according to program. 

The functional organization of OSFA has both disadvantages and advantages 

relative to the design and development of a QC system for the Pell program and other 

student aid programs. The principle disadvantage is the absence of a programmatic 

emphasis that cuts across all divisions in OSFA and identifies QC measures and 

standards for individual programs, especially for processes that cut across division 

lines. Therefore, it is important that a quality control subsystem link division s 

together . The design of a Pell QC system must take into account this serious 

N 

limitation. 
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The principle advantage of the organizational structure for the QC system design 
activity is that the functional emphasis makes it possible to develop QC modules that 
view the various operations as an integrated system and can thus assess performance 
across program areas. As the QC design effort continues, it will be important to 
evaluate which QC modules can perform functions across programs and yet provide the 
necessary linkage across divisions for each student aid program. 
OSFA Management 

The Deputy Assistant Secretary for Student Financial Assistance has recently 
proposed a goals and objectives management system for OSFA that is currently being 
implemented. This system provides a potentially sophisticated ar.d effective process 
for managing OSFA programs. The intent of the system has been characterized as 
follows: 

To provide improved means of resource planning and management, OSFA 
personnel have developed and implemented at a management level the 
goals and objectives system, to assist personnel in understanding the 
numerous activities that must be accomplished. Information can be saved, 
modified, and presented to assist key personnel- in meeting objectives in a 
timely manner and in developing resource plans. 

The primary objective is to provide OSFA with a means of planning, allocating, 
and tracking the use of resources easily in a changing environment with respect to 
OSFA's key goals and objectives. The system organizes OSFA goals through the 
specific steps required for implementation. Specifically, this is defined as follows: 

• Goals: Reflect OSFA priorities. 

• Objectives: Identify division responsibility. 

• Activities: Identify branch responsibility within division* 



D. Kerr, ,f Goais/Objectives Management System," OSFA, Division of Systems 
Design and Development, April 27, 1982, page 2. 
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• Tasks: Identify section within division. 

• Subtasks: Identify unit responsibilities. 

• Steps: Identify personnel responsible for specific steps in the implementa- 
tion process. 

The first step in implementing this system has been accomplished. Divisions 
have identified goals through steps as they relate to division responsibilities. The first 
round of goals through steps are currently being entered into an automated TRS-80 
system. The system itself is intended to result in: 

• Improved resource allocation 

• Improved office-wide communication 

• Improved long-term planning 

• Improved flexibility 

According to current plans, the system will provide OSFA with a primary means 
of planning, allocating, and monitoring resources. The Division of Systems Design and 
Development has been given responsibility for implementing the new system. It will 
eventually provide each division with Gantt charts for divisional objectives through 
steps. After review, this system will become an integral part of the OSFA 
management system. 

The goals and objectives system provides OSFA with a significant vehicle for 
improved QC, especially if program specific information flow and program responsi- 
bilities are identified. In fact, the information currently entered in the system can be 
used to develop program specific QC standards for Pell and other OSFA programs. 
QC Issues 

The major QC issue at the management identified level in OSFA as a result of 
this review is that program management can break down because of the functional 
organizational structure in OSFA. The design of a QC system for Pell or other student 
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aid programs should identify program specific linkages across OSFA divisions* The 
goals and objectives management system currently being implemented in OSFA 
provides a framework for identifying these programmatic linkages across divisions and 
identifying QC standards for performance in critical areas. This system should be used 
as a framework for developing a management level QC system for the Pell program 
and other student aid programs. Each system should include attention to: 

• Critical points in the policy cycle for program policy, planning, and 
regulations for each program 

• Critical points in the delivery cycle for each program 

ANALYSIS OF THE OPERATIONS LEVEL IN OSFA 

The following discussion considers the information resources and QC needs at the 
level of operations or program delivery for the Pell program. Two divisions are not 
covered in the following analysis: Division of Quality of Assurance (DQA) and Division 
of Systems Design and Development (DSDO). DQA concerns are considered throughout 
this manuscript, while the goals and objectives system in DSDD is addressed in the 
above section. In addition to a discussion of the other divisions, the role of OSFA 
Regional Offices is considered. The specific topics considered in this section are: 

• Division of Policy and Program Development, Pell Grant Branch 

• Division of Program Operations, Peil Grant Branch 

• Division of Certification and Program Review 

• Division of Training and Dissemination 

• OSFA Regional offices 

Division of Policy and Program Development, Peil Grant Branch 

The DPPD Peil Grant Branch is one of three branches within the Division of 
Policy and Program Development. This office is responsible for the development of 
program regulations and budget packages as a part of the regular policy and budget 
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cycles within the Department of Education. The branch is divided into two sections; 
policy and analysis. The policy section is in charge of regulation development, while 
the analysis section generates a budget package with policy options that goes to the 
Secretary, conducts periodic analytic studies, and supports the application processing 

4 

contractor. 

As with its counterparts in DPPD, the primary function of the Pell Grant Branch 
is to translate the general policy direction of the Pell Grant Program given it by the 
Office of the Secretary into specific tasks needed to operate the program. Budgetary, 
legislative, and regulatory changes are formulated where needed and become part of 
the regular cycle for setting the program parameters for operation. Unlike the other 
branches within the division, however, the Pell Grant Branch also controls the 
processing of application forms for the program. This function adds additional 
responsibility of the branch for developing the application forms and ensuring the 
efficient processing of the forms and notices of eligibility. It also blurs the distinction 
between the roles that policy development and operations play in the overall delivery 
of the program. 

QC Procedures 

In terms of the branch's policy development function, the branch chief outlines a 
number of activities which must be accomplished during a particular month and a 
number of ongoing activities which the staff can be working on. This list is gone over 
with each section chief so that priorities may be set. The list also allows for a way of 
reviewing which tasks s et done and which do not. This system controls the internal 
workings of the branch given the set of tasks imposed on it by the senior policymakers 
in the Department. However, the tasks given to this branch, like those of other 
branches in the division, are highly variable and change with the political situation. 
This means that the list of activities and their priorities must be amenable to change. 
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As faf as the integrity of the program's delivery is concerned, the branch 
function is the front end of the process. The policy development aspect of the branch 
does not have any QC procedures per se . The branch can only maintain quality to the 
extent it can foresee how the delivery system will work. The basis it uses for any kind 
of review of activities over the year is through the overall policy process. 

The processing oversight function of the branch requires a different type of 
management approach. The branch's role is oversight and not production of the work. 
The branch performs the same type of tasks as those in the policy development 
function, that of translating policy into specific operations. The branch must then go 
one step further and ensure that specific tasks are performed correctly. A main 
function of the Branch is to respond to ad hoc requests for information and analysis 
from OPBE, OMB, and Congress. 

To monitor the Pell Grant processing contract, the branch receives weekly and 
monthly progress reports from the processor detailing several indicators of processing. 
These reports compare this year's production with last year's production with the 
contract standards. These reports reflect the production aspect of the processing and 
not so much the quality of the processing. To the extent that effective management 
indicators can be quantified, the reports adequately reflect the status of the 
processing. 

Information Resources 

In order to carry out its policy development function of generating the necessary 
budget projections, the branch relies on a cost estimation model that uses as us data a 
file of applicants and recipients. The model has the capability of simulating a set of 
program parameters and estimating a total cost for the program. The list of 
objectives for each month allows members of the branch to know what the activities 
will be and allow section chiefs to set priorities based on the most reliable information 
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at the time. The branch receives information through many sources, but one formal 
method is the branch chief meetings with the division head. Information from these 
meetings on policy directions is passed to the respective branches for implementation 
and information on the feasibility of the policy guidelines and then passed back 
through to the policymakers. 

The weekly and monthly management reports from the processing contractor 
provide the main source of information on the condition of the processing. In addition, 
ad hoc reports on specific aspects of the system can be requested from the contractor. 

Information Needs 

Due to its nature of having both policy development and operation functions, the 
branch has a wide range of needs to ensure proper integration of functions. First, the 
branch needs better information between the back end of the process, that is, from the 
Division of Program Operations. DPO has control over the institutional fund 
disbursement and accounting. The policy implementation done by the Pell Grant 
Branch has a direct impact on the fund control and allocation of funds to institutions. 
Second, since the branch has the primary responsibility for implementation of poiicy, 
the branch needs better information on the general direction of that policy with as 
much lead time as possible. This may not be possible due to the unpredicteMeness of 
political situations, but the branch must have a full amount of information as soon as 
possible. 

Division of Program Operations, Pell Grant Branch 

The Division cf Program Operations is charged with the actual carrying out of 
ED operations for th» Pell Grant, Campus-Based, and GSL programs. ED's operational 
role involves largely the disbursement of Title IV funds to institutions and the 
reconciliation of fund expenditures at the institutional level. Because this division 
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essentially controls the purse strings for the Title IV programs this is a critical area 
for QC. 

The DPO Pell Grant Branch is responsible for the disbursement and monitoring of 
Pell Grant funds to institutions* The branch has three sections: Fund Control and 
Disbursement (FC&O) Systems, and the Alternate Disbursement System (ADS): 

• The fund control and disbursement section has two units* The Fund Control 
Unit is responsible for accounting functions, including updates to the 
universe file. The Disbursement Unit processes the progress reports and 
validation rosters submitted by the institutions for finding authorizations 
and serves as the liaison with institutions. Essentially, they monitor the 
flow of funds to institutions throughout the year. 

• The Systems section maintains the data base on all institutions receiving 
Pell Grant funds and tracks the number of SARs submitted by each 
institution. This is done through the Program Information and Monitoring 
System (PIMS). 

• The ADS section handles the direct disbursement of Pell Grant funds to 
students for those institutions on the ADS system. 

QC Procedures 

The Pell Grant Branch currently uses the following QC procedures to monitor 
operations: 

• The PIiMS staff refers error -prone and potentially fraudulent schools to the 
Division of Certification and Program Review, 

• Through the PIMS subsystems, this branch identifies discrepancies between 
SARs and the accompanying progress reports submitted by institutions. 

• The Pell Grant Branch tracks transaction processing errors made by 
institutions on progress reperts and other submissions. Schools which make 
excessive transaction errors are contacted. 

• The branch has a system to identify late report filers and institutions that 
continue to overestimate funding requests. 

• Validation roster submissiohs are monitored* The branch can freeze funds 
for institutions which fail to return a/roster after 60 days. 

• The PIMS system uses a reasonable test on institutional fund requests. 
Each request for funds is compared to the expected total funds to be 
allocated to that institution. 



3-14 



55 



Information Resources 



The Basic Grant Branch contains infocmation sources available for possible use in 
a QC system. PIMS produces a wide range of reports including ageing reports, 
unreasonable reports, receipts, control reports, and error reports. PIMS is comprised 
of five subsystems: the Annual Initialization Subsystem, the Universe Subsystem, the 
Progress Report Subsystem, the Accounting Subsystem, and the Student Eligibility 
Report Subsystem. The information available and the function of each is described as 

\ 

follows: 

• The Annual Initialization Subsystem is used to perform all functions which 
are necessary for beginning processing of a new school year. This process 
facilitates a timely mailing of authorization information to eligible insti- 
tutions. 

• The Universe Subsystem maintains the master universe file of institutions 
which are eligible for consideration for participation in the Pell Grant 
Program. The universe file contains demographic, classification, enrol- 
lment, participation status, obligation, expenditure, and recipient count 
data as well as a variety of monitoring information. 

• The Progress Report Subsystem performs all of the collection, editing, and 
correction of data submitted to ED on the progress report. The informa- 
tion reported in this document is evaluated within the system to validate 
institutions' requests for additional BEOG funds. Reports are produced 
regarding errors on data submitted, count, and amounts of excessive 
(unreasonable) requests for funds by institutions. Based on this data, 
various statistical reports, ageing reports unreasonable reports, error 
records, progress reports, and Student Validation Rosters are produced, as 
well as various control reports. 

• The Accounting Subsystem performs all the functions related to the actual 
obligation of BEOG funds to institutions. The subsystem updates the 
master institution universe file with obligation information and produces 
letters and institutional/state/regionai reports on obligation of funds. 
Also, various error and control reports are produced to monitor funds to 
institutions. The subsystem passes obligation transactions to the ED 
finance group to match the disbursed funds. Various reconciliation reports 
exist to monitor discrepancies in obligation and disbursement of funds. 

• The Student Eligibility Report Subsystem performs all activities related to 
recipient-level fund control in Pell PIMS— the SAR returned to ED by the 
institution to the means through which development of data bases on the 
Pell recipient level is accomplished. SAR data reported are collected, 
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edited, and verified against application data collected by the Peil applica- 
tion processor system. Various control reports, error reports, and summary 
fund control processing and statistical reports are produced. 

Other sources of information exist for QC outside o$ PIMS, such as: 

• Processing, disbursement, and reconciliation data from ADS 

• Information on phone and letter inquiries made by institutions. This data 
may not be in useable form at present but are potentially a resource. 

In addition, the branch is currently developing a program cost projection model to 

estimate program costs based on the most up-to-date disbursement information. When 

completed the model should provide additional QC checks in the disbursement process. 

Information Needs 

In interviews the Pell Grant Branch expressed the need for uniform quality 
control measures for contractor data entry error. Greater system integration with th * 
processor is needed to catch error and to ensure that students ve being paid with the 
correct 5AR. This branch also needs better integration and more feedback from the 
Program Review* Branch in the areas of institutional eligibility, audit results, and 
authorizations of money to institutions approved during a program review* 
Division of Certification and Program Review 

DC PR Certification Branch 

The Certification Branch is responsible for the certification of new institutions 
entering the Title IV programs. The new institutions may be either new to all the Title 
IV programs or adding a new program (e.g., Pell Grants to an existing Campus-Based 
school). The Certification Branch is the third step in the ED approval process for new 
institutions, after accreditation and eligibility reviews. The branch consists of three 
sections; Coordination, Administration Analysis, and Financial Analysis. Each plays a 
specific role in the Certification Review Process. 

This branch is also responsible for recertification of institutions that have 
changed ownership. This review is the most extensive type performed by the branch. 
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Since there is a track record available to analyze, the branch will determine any 
obligation to ED from the old ownerships for operations from Title IV programs, if 
necessary, Regional Office Program Reviewers will visit the institution and review 
records there. 

The Certification Branch currently uses the following QC procedures to monitor 
their operations: 

• Supervisory review of the Certification Reviews, including a review of a 
checklist to ensure that all steps have been performed 

• Branch chief review of "cover sheet" on which each reviewer has signed off 
if the Certification Review is not proceeding on time, with the Branch 
Chief following up with the staff involved 

DC PR Validation Branch 

The Validation Branch is responsible for development and review of the yearly 
Pell Grant validation procedures for all actors (processor, institution, and students.) A 
notice of proposed rule making has been issued to add a Campus-Based validation 
process, but currently validation is performed only on Pell Grants. Additionally, this 
branch handles all ADS validations (2,600 per year) and institution referrals of problem 
cases (100 per year). Some cases involve collection of overpayments referred to ED 
regional offices, and about 20 cases per year are referred to the Inspector General for 
prosecution. The Validation Branch currently uses the following QC procedures to 
monitor their operations: 

• Supervisory review of ADS validations performed by the branch prior to 
closing of cases (one person handling all institution referrals) 

• A weekly report of work completed within by each staff member, used for 
internal monitoring purposes 

• Time standards for completion of annual handbook and selection criteria, 
when ADS validations are required within 30 days of receipt 
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DC PR Audit Review Branch 

The Audit Review Branch is responsible for review and disposition of audits 
performed on institutions for the Pell Grant and Campus-Based programs. (Audits are 
not presently required for GSLP.) The branch processing of audits is the same for both 
programs. The audits are performed by outside CPAs or internal or state auditors. 
After completion of the audit, the report is submitted to the Regional Office Inspector 
General for Audit (IGA). (The branch has no regional personnel.) The Inspector 
General checks the report for completeness by the Audit Guide and forwards it to the 
branch. The branch consists of three sections, each serving specific geographic 
regions. Processing of the audit is performed in the appropriate section, and 
institutional agreement of final disposition and any corrective action is obtained. At 
that point the case is closed. The Audit Review Branch currently uses the following 
QC procedures to monitor their operations. 

• Cases are required to be closed within six months of receipt. Additionally, 
each reviewer is required to close 13-15 cases per month for an annual 
total of 16 1 cases. 

• The Section Chiefs and Branch Chk review all cases. Deficiencies noted 
will be pointed out to the staffer who chen processes the cases. 

• This process serves as QC on the audit itself. The regional IGAs review the 
auditors' work papers, as required. 

DCPR Program Review Branch 

The Program Review Branch is responsible for monitoring institutional and 
lender performance in the Title IV programs. Program Reviews are very similar to 
audits just outlined but concentrate more on program operations and less on annual 
financial reviews. Additionally, when a Program Review is performed, all programs 
are examined. The reviews are performed by Regional Office staff but scheduled by 
the Central Office. This process focuses the Program Review process on the problem 
schools. (Approximately 1,100 institution and 900 lender reviews are performed 
annually.) 



After completion of the review field work (usually three to five working days) 
the reviewer drafts a report outlining findings and suggesting corrective actions. A 
number on a scale from I to VI is assigned (I being clean, VI being referral to Inspector 
General). When institutional agreement with findings is obtained, the report becomes 
final, and the case is closed. The following QC procedures are currently used by the 
Program Review Branch: 

• All cases are reviewed by a supervisor in the Regional Office. (In some 
cases the supervisor participates in the field work.) All cases assigned a 
"high 3" or above are reviewed in the Central Office. 

• Program review findings are compared with audit results for consistency. 
Follow-up reviews are also scheduled in many cases to monitor imple- 
mentation of corrective actions. 

• Institutional reviewers have a standard of one and a half closed cases per 
month; lender reviewers have a standard of two cases per month. 

• Timing standards for cases require adherence to the central office 
schedule— 10 days (after completion of field work) for the draft report and 
30 days for the final report. 

DC PR, Information Resources 

The Division of Certification and Program Review has the following information 
sources available for possible use in a QC system: 

• The School Monitoring System (SMS). This system is not implemented and 
is still in the developmental stage. In our interviews with the DCPR 
branch chiefs, we were told that, when implemented, this system will 
contain a data base of ail institutions participating in the Title IV 
programs, as well as the results of all audits and reviews performed bv 
DCPR./ " 7 

• Informat ; on on all validations performed by the Validation Branch 
(primarily ADS validations) 

• Information pertaining to all program reviews and audits performed by 
DCPR. Until SMS is implemented this information will have to be 
retrieved manually from the case files but is a potential resource. 

DCPRj Information Needs 

All the DCPR Brancn Chiefs interviewed stated that current information 
resources are not adequate to meet their needs. They all acknowledged they are not 
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abie to disseminate adequate information on the results of their monitoring efforts to 
other units of OSFA. Additionally, iiie Validation Branch desires an automated 
information system to track ADS validations and provide reports on findings. (This 
need will become acute if 100 percent validation is implemented and the branch 
becomes responsible for 35,000 validations.) 
Division of Training and Dissemination 

The Division of Training and Dissemination (DTD) is responsible for the student 
financial aid and fiscal officer training programs as well as preparation and dissemina- 
tion of handbooks, update materials, and any other pertinent information needed in the 
field. They also handle inquiries from students, parents, financial aid officers, high 
school counselors, and the Congress. DTD does some in-house training but would like 



to have more responsibility and input in this area* Their training responsibilities span 
all of the Title IV programs. 
QC Procedures 

The main QC procedures within DTD focus on the training programs and the 
letter inquiries. Through a contractor DTD sponsors training programs at over 400 
sites per year. In addition, 2,000 to 5,000 letters per week are either received or 
referred to this division. The principal reports received by DTD which may be used for 
QC are: 

• Student/consumer complaint reports supplied through the Biospherics 
contract which conducts a student/consumer complaint survey three times 
per year 

# Periodic reports from the Division of Programs Operations on the most 
common student errors dealt within that division 

• Monthly reports from the training contractor giving the status of the 
training program including evaluation results on the effectiveness of the 
training sessions 

# Control reports on the number of inquiries answered each week 
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Information Resources 

DTD receives updates on all policy and regulatory changes in o^ier to dis- 
seminate these to financial aid officers and others in the field. During the annual 
development of training materials they receive information from several divisions 
(DCPR, DPO, DPPD, and DQA) on policy changes, changes in validation and program 
review procedures, and in funding and disbursement procedures. They also have access 
to a wealth of information through the inquiries received from the field and those 
which come in from I AOs via the area desk representatives. One other source is the 
student/consumer complaint survey. 

Information Needs 

Our interview did not reveal precisely how DTD uses the information it receives. 
However, much of it seems to be used in the development of training curriculum and 
production of handbooks such as the Counselor's Handbook and the Q and A Handbook . 

DTD feels QC in their division could be enhanced in two areas: 

• Use of the large volume of inquiries as a QC monitoring tool 

• Obtaining evaluation data from all other divisions on their in- house training 
needs 

Currently, little or no analysis of letter and telephone inquiries other than 
quantitative tabulations of volume is being done. An analysis and categorization of 
inquiries according to content would help identify areas of greatest error and 
confusion. This information would contribute to the improvement of training and 
could be disseminated to other divisions for their use. 

DTD would also like to develop some sort of monitoring mechanism including a 
routine information exchange to identify training needs in other divisions. At present 
DTD is not included in the planning and development of training programs in other 
divisions, a role they feel is relevant to their capabilities. An example given during 
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the interview was that of training the area desk representatives in DPO. Currently, 
this is contained within the division since the training would focus on system errors. 
These kinds of system and procedural errors occurring within that division are not 
routinely shared with other divisions. 
QSFA Regional Offices 

There are 10 OSFA Regional Offices located throughout the United States. Two 
Regional Of f ices— Atlanta and Philadelphia— were visited for this study. These offices 
currently perform the following functions: 

• Institution, lender, and guarantee agency program reviews 

• Training and dissemination 

• GSL and NDSL claims processing 

• Collections of GSL and NDSL claims (only in Atlanta, Chicago, and San 
Francisco) 

This process and QC procedures for program reviews is detailed in the OCPR section. 
With the exception of the three offices involved in collections, this is the major 
function performed in the regions. The collection process is currently split, with some 
accounts being referred to collection agencies. Therefore, the collection function in 
the region includes: 

• Monitoring the collection agency 

• Calling students to urge repayments, etc. 

• Reconciling exceptions (such as checks posted to suspense account) 

• Write-off of balances, as appropriate 

• Updating the GSL system for such things as address changes, status 
changes 

QC Procedures 

• Procedures used in monitoring of collection agency 

Audit transactions and monthly commission invoice 



On-site monitoring which includes listening to phone calls and obscur- 
ing each receipt processing 

Review of accounts to ensure that standards detailed in the RFP are 
maintained 

• Procedures used in internal QC for collections 

Monitoring of telephone calls 

Testing of telephone calls to collection personnel to measure 
response 

Supervisory review of keypunched data to the GSL system and 
monthly spot checking of outputs to inputs 

Information Resources 

Each region maintains information pertaining to all program reviews performed. 
In addition, the Regional Office IGA maintains records of audits received. 

In Atlanta the collections function (Division of Accounts Management) maintains 
information on all GSL and NDSL accounts for Regions I to IV they are responsible for. 
They also have transaction runs, ledgers, etc., from the GSL system (plus the ability to 
query the systems). 

Information Needs 

The region personnel we met stated that current information resources are 
inadequate to meet their needs. They desire: 

• An automated review in information systems that could: 

Report on trends, etc. 
Track recoveries 

Prepare needed activity reports for control office 
Monitor performance standards 
Compare reviews to audit findings 

• Modifications to GSL system that 

Will allow automated audit of monthly commission invoice 
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Automatically sample accounts for detailed review and comparison to 
standards 

Make other corrections to specific errors and lack of controls they 
see in the system 

Other Issues 

Region personnel with whom we met raised additional points regarding the 
current monitoring process. Specifically, they felt a shift in focus was needed for 
institution program reviews to allow in-depth investigation where major accounting 
problems or fraud and abuse were evident. An additional concern was raised with 
lenders in that no outside audit is currently required and lender reviews are only able 
to cover 10 percent of all active lenders on an annual basis. 

This issue is basically an ED policy issue, covered in depth in our deliverable on 
the site visits. It is mentioned here because it is the major concern to come from our 
visits and because, if appropriate, we could consider it during design of the institution 
and GSL monitoring systems discussed in the next chapter. 

CONCLUSIONS 

The three-tiered analytic framework provides an effective lens through which to 
view the QC needs and information resources for the Pell Grant Program. The types 
of QC needs identified in the functional analysis clearly vary by level of organization 
for OSFA student aid programs. There is an obvious inter dependency of information 
sources, both between levels from management to operations and between 
units/divisions and branches at the operations level. 

At the policy level, QC concerns reiate primarily to the quality of overall 
program delivery. Outside of OSFA, in ED and OMB there is a concern that payment 
error in the Pell program be assessed on an annual basis. Additionally, analysts in 
OPBE are interested in receiving timely analysis of policy options for corrective 
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actions involving legislative or budget action. These same concerns are apparent 
within OSFA, along with an accurate awareness that critical parts of the policy cycle 
(planning, budgeting, and regulation) and program delivery (application through recon- 
ciliation be performed on a timely basis. This suggests that a policy level QC 
management information system is a desirable development, especially if such a 
system provides: 

• Policy analysis for corrective action alternatives 

• Early warnings for critical problems with program delivery or policy 
development 

• Analysis of the effects of critical problems with the policy or delivery 
cycles 

• Periodic assessment of overall payment error in the delivery of Pell Grants 
to the management level. Quality control for the Pell program is hindered 
by the lack of a program emphasis in the formal organization structure of 
OSFA. The functional organization with the office makes programmatic 
developments more difficult because responsibilities for the program are 
spread across divisions with no formal linkages between critical program 
functions. Therefore, the timely flow of information between functional 
units can be inhibited. The analysis needed by another division may take 
second priority behind performance of key activities within the division. 

The goals and objectives system currently being implemented by the Division of 

Systems Design and Development has the potential of overcoming this key problem 

with program management if appropriate QC enhancements are developed. If a 

systematic review and analysis of the data entered in the system were undertaken it 

would be possible to develop a QC monitoring and tracking system for critical points in 

the program delivery and policy cycles. Such a system would require: 

• Development of critical paths and flow charts for the Pell delivery cycle 

• Analysis of information in the goals and objectives system to identify early 
warning problems 

• Development of a monitoring system that reports early warnings for 
deviations from the critical path for the delivery and policy cycle 
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At the operations level, QC issues are more abundant. Branches and divisions 
have developed their own QC procedures, as needed, usually on an ad hoc basis. As a 
result, there is a wide variation in the awareness of staff within branches about QC 
issues. Some branches have developed fairly sophisticated approaches to QC, while 
others are confronted by basic problems. The branches are critical to QC in two 
essential ways: 

• Most of the information needed for policy- and management-level QC 
systems must be generated or already exists at the operational level in 
OS FA divisions and branches. 

• There are numerous places at the operational level where QC subsystems 
could be developed that would: 

Provide QC information for management and policy 

Provide linkages between functional areas (across divisions) 

Identify corrective action options for critical error-prone areas 
within divisions 

The OSFA goals and objectives system is one information resource that gathers 
data directly from operating units and has direct applicability to the design of a QC 
system. In addition, there is potential to develop QC subsystems in the error-prone 
areas analyzed in the Stage One study— e.g., for applicants, processor, institutions, and 
OSFA. One advantage of the functional organization structure in OSFA is that it is 
possible to identify possible host divisions for subsystems for each of these actors. 
The next chapter presents a preliminary design for a modular QC subsytem that 
interfaces with the functional organizational structure in OSFA. 
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CHAPTER ♦ 
PRELIMINARY QC SYSTEM DESIGN 
FOR PELL GRANT PROGRAMS 

This chapter presents a preliminary design for the Pell Grant QC system. A 
modular QC system design is presented, with six basic subsystems. The possible 
integration of the other student aid programs— Campus-Based and GSL—into the 
overall design is also considered. Before a discussion of each subsystem, an overview 
of the proposed Pell Grant QC system is presented noting the interface between the 
QC system and the OSFA hierarchy. 

QC SYSTEM OVERVIEW 

The preliminary QC system design identifies modular components that interface 
with the current delivery system 'based on the functional analysis of the QC needs, 
resources, and systems in OSFA. It is now apparent that: 

• Existing data sources and management processes for the Pell program 
provide a basis for the design of an internal QC system. 

• It is possible to design a QC system with modular components that can be 
developed incrementally. 

The organizational perspective of the Pell Grant QC system is reconsidered in 

Figure 4-1. Given the current operating environment in OSFA, it is possible to design 

an integrated Pell Grant QC system through a strategy of providing technical support 

to OSFA divisions to facilitate QC efforts and systems currently under development. 

According to this view, the modular systems in the overall QC system would be: 

• A QC Subsystem for Assessing Annual Overall Payment Error in the Pell 
Grant Program to provide external assessment of error in the Pell delivery 
system, operated by the Division of Quality Assurance 
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AN ORGANIZATIONAL PERSPECTIVE OF THE PELL GRANT QC SYSTEM IN OSFA 
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• An Institution Review QC Subsystem to provide systematic reporting from 
the Pell Grant component of the Schools Monitoring System (SMS) operated 
by the Division of Certification and Program Review 

• A Student/Disbursement QC Subsystem to provide technical support to the 
Pell Grant Branch, Division of Program Operations, in the use of student 
error -prone profiling to build additional checks on the institutional dis- 
bursement process 

• A Processor /Vendor QC Subsystem to provide technical assistance in 
implementing QC procedures for the Pell Grant processor, probably housed 
in the Pell Grant Branch, DPPD 

• An OSFA Goals and Objectives Monitoring QC Subsystem to provide 
technical support to the Division of Systems Design and Development in 
the implementation and enhancement of the OSFA goals and objectives 
process 

• A Policy-Level QC Management Information System to provide monitoring 
of OSFA operations and policy analysis for policy relevant variables for the 
Deputy Assistant Secretary for Student Financial Assistance, housed 
administratively in the Division of Policy and Program Development 

The first of these subsystems would provide OSFA with a capacity for detecting 
and documenting overall payment error on an annual basis. It would provide an 
external measure of delivery system performance, much as the Stage One study 
measured total error in the Pell Grant program. Such a subsystem would provide only 
a limited capacity to analyze corrective action options or assess the effects of such 
options once they were selected and implemented. This requires a QC system that 
provides a mechanism for controlling or at least coordinating all phases of organiza- 
tional activities which have a bearing on the quality of the delivery of student aid. 

In combination, the other five subsystems provide a system for internal control, 
as well as for QC. Four of the modular subsystems would provide a capacity to 
statistically monitor performance of a major actor in the Pell delivery system: the 
students, processors, institutions, or OSFA. As a result, the success of specific 
corrective actions could be tracked and assessed through these subsystems in combina- 
tion with the QCMIS, which would provide a policy-level analytic capacity for OSFA. 
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Thus, the QCMIS would utilize the outputs of the other subsystems in order to analyze 
policy (correcting action) options and to assess the effects of selected options* 
Eventually, if the internal QC system proved to be an effective way of monitoring and 
correcting error, the external QC subsystem would be less important* In fact, it might 
be possible to do the same analyses with the combined internal QC system. 

These subsystems constitute targets of opportunity for technical assistance as 
well as for modular QC subsystem development. Each component of the subsystem is 
explained more fully in the following section. The following discussion covers five 
basic topics for each proposed subsystem: 

• Statement of the problem 

• Existing procedures 

• Location of the subsystem 

• Description of the subsystem 

• Benefits to OSFA of implementing the subsystem 

QC SUBSYSTEM FOR ASSESSING ANNUAL OVERALL PAYMENT ERROR 
Statement of the Problem 

In addition to having management information from functional areas within 
OSFA related to the performance of the Pell Grant delivery system as an input to 
policy deliberations, it is important that policymakers have an annual measure of 
overall error in the delivery of student aid. Therefore, as a part of the QC system 
design, it is essential that a system for assessing the overall payment error in the Pell 
program be included* The purpose of the system would be to provide an annual 
measure of overall payment error in the Pell program. 
Existing Procedures 

Currently, OSFA has no ongoing system for measuring payment on error in the 
Pell program and providing an external check on the performance of the entire system 



on an annua! basis. While DCPR performs a review function for all student aid 
programs, it does not gather the type of evaluative information required to monitor 
system performance. The Stage One QC study was the second attempt by OSFA to 
undertake such an external monitoring effort. The purpose of the external system for 
assessing error in the delivery system is to provide an ongoing system for routinely 
(probably annually) collecting evaluative information on how the Pell delivery system 
is performing. 
Location of the Subsystem 

The actual position of the external subsystem in the overall QC system design is 
easy to explain. It would be external to the QCMIS but provide reports directly to it. 
The location in OSFA is more difficult to specify. The function may be performed by 
DQA. 

Subsystem Description 

The Stage One study provided a comprehensive analysis of error in the Pell Grant 
delivery system. Overall error in the program was measured and the greatest source 
of error identified. In the future it may be desirable to conduct external QC studies, 
similar to the Stage One ;tudy, which collect data beyond the capability of an 
"internal" QC system and provide an overall measurement of program performance. It 
is possible to conceive of this process as an ongoing system for assessing payment 



error. 



Future external QC studies, of course, would not necessarily have the same 
design, scope, or study population as the Stage One study. For example, it is possible 
that OSFA regional personnel can collect data on the effects of validation, rather than 
using the independent field interview method used in Stage One. A combination of 
approaches may also be possible to test the validity and accuracy of different 
collection methods. 
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The purpose of this area of technical assistance would be to provide the Division 
of Quality Assurance with specifications for an ongoing system for assessing payment 
error. Therefore, the emphasis of this system design effort would be on the mechanics 
of conducting this type of data collection and analysis. The study design would cover 
the following areas: 

# Sampling : Various sampling designs and procedures would be considered* 
The strengths and weaknesses of each approach would be thoroughly 
considered. Various sampling and study bias issues such as estimating, 
sampling error, sample size, survey nonresponse estimation, and sampling 
techniques for limiting data collection costs would be included. 

# Data Collection : The costs and benefits of three methods of collecting 
data could be reviewed: field interviewing (as was used in the Stage One 
study), telephone interviewing, and mail surveys. The system design would 
give a step-by-step guide for implementing each survey method, would 
provide a time line for each, and would analyze staff and other resource 
needs. Problems specific to interviewing aid recipients and collecting data 
from student financial aid files would be considered in detail. 

# Study Instruments : The system design would detail various approaches in 
developing interviewer-administered questionnaires, self -administered 
questionnaires, and protocols for telephone interviewing. Pilot testing of 
survey instruments and the OMB/FEDAC forms clearance process would 
also be considered in the overall design. 

# Data Base Development : The design would cover procedures and resources 
required to clean, edit, and enter survey data into a data file. 

- ^ Data Analysis : The subsystem design would review possible statistical 
analysis techniques applicable to assessment— from simple descriptive 
statistics to complex multivariate statistics, analyze the merits of avail- 
able statistical software packages, identify hardware needs, and identify 
staff and training needs. 

Benefits to OSFA 

The primary benefit to OSFA of implementing the subsystem for assessing error 
in program delivery would be that it would provide an ongoing system for monitoring 
the performance of the overall Pell Grant delivery system. 
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INSTITUTION REVIEW QC SUBSYSTEM 
Statement of the Problem 

Postsecondary institutions in the Regular Disbursement System (RDS) play a 
major role in the cyrrent Pell Grant delivery system. They verify student eligibility, 
calculate the amount of award, disburse the award to individual recipients, and 
perform validations on students selected by OSFA. These activities could be routinely 
monitored as part of the institution review subsystem of the Pell Grant QC system. 

Current institution monitoring activities are detailed in the following section* 
However, there is no system implemented currently that can quantify institutional 
error , particularly as it relates to the Pell Grant Program. It is necessary to quantify 
institutional error by dollar amount, type, and number of occurrences. This will assist 
OSFA in planning and implementing policy changes, procedural changes, and corrective 
actions designed to reduce error. It will also furnish OSFA with a more complete list 
of "problem schools." Training and technical assistance could then be targeted to 
those institutions. 
Existing Procedures 

Institutions are currently monitored by the Division of Certification and Program 
Review. Monitoring activities include program reviews and audits, which are 
performed by independent accountants on an annual or biannual basis (institution 
option). The independent accountants are engaged by the institution, and the report is 
furnished to ED and the institution. The Regional Office of IGA receives the audit 
report and reviews it for compliance to the Pell Grant Audit Guide and completeness. 
The report is t) l e t » pi oces s ed by the Audit Review Branch (ARB) in Washington. After 
completing this review the ARB reviewer submits a letter to the institution outlining 
the agreed findings and prepares a coding sheet detailing the findings for the School 
Monitoring System (SMS). 
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The SMS is currently in the developmental stage. A file of institutions to be 
monitored does exist, and ARB has developed a list of audit deficiency codes. These 
are entered into SMS as audit reviews are completed. Currently, SMS cannot 
manipulate or process these data. No outputs or reports are produced. After 
implementation, this problem should be remedied. 
Location of the Subsystem 

The institution review subsystem would provide an interface point between SMS 
and the Pell Grant QC system. It would be housed in DCPR. The QC subsystem will 
not require separate data entry since output from the SMS would be an input to this 
QC subsystem. 
Subsystem Description 

The proposed subsystem would have the following information flow: 

• As audits and program reviews are completed in the field, Regional Office 
personnel would enter findings in the subsystem (school, deficiency codes, 
dollar values, staff identification status). 

• These would be updated as the cases are reviewed in Washington, and 
institution agreement obtained. This would provide the basis input to the 
QC subsystem. 

• The QC subsystem would utilize predefined QC measures and standards to 
monitor institutional performance, using the reviewed institutions as the 
basic sample. 

• The subsystem would produce the following types of outputs 

Listing of schools with specific error (such as improper validation) 

Reports on review results including overdue cases to monitor 
performance against-standards 

Statistical analysis of findings (such as trends) 

if the institution validation option of IRS 1040s for ail recipients is implemented, 

information on the results of this option would be integrated into the Institution 
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Review Subsystem, The level of integration could include a tracking subsystem for 
IQkO validation or a findings reporting subsystem. It would also be possible to develop 
an institution QC manual as part of this paper as many institutions are not automated, 
and developing the telecommunications between automated institutions and OSFA 
regional offices may prevent or delay integration of this process. 
Benefits to OSFA 

Major benefits of this activity will flow mainly to DCPR but will also assist 
other units of OSFA in the operation of the Pell Grant delivery system. These benefits 
include: 

# Measurement of reviewer performance against standards for timeliness and 
number of cases completed 

* Early notification of overdue audits and program reviews 

# Ability to respond to queries on specific schools 

a Reduction in paperwork generated by ARB reviewers and program 
reviewers 

# Quantification of institutional error by; 

Amount 
Type 

Number of occurrences 

• Ez.se in dissemination of information to specific OSFA units regarding 
institutional error 

• Identification of areas x procedures on which to focus DTD training 
modules 

STUDENT/DISBURSEMENT QC SUBSYSTEM 
Statement of the Problem 

The student/disbursement subsystem would both monitor performance against 
standards and determine error levels. Information currently collected in OSFA could 
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technically be better utilized to monitor QC measures and standards. Specifically, the 
purposes of this subsystem would be: 

• To identify and quantify the amount of student application error, the types 
of errors, and the prevalence of such errors 

• To utilize these data to reduce the errors or the tendency toward error in 
the fund disbursement to error-prone institutions and students 

• To identify institutions with unreconciled disbursement records and initiate 
required corrective actions or program reviews 

• To assist in the development of policy changes and the necessary 
corrective actions to reduce and better control both the amount and the 
frequency of various types of application errors 

Current Situation 

The basic elements of a student/disbursement subsystem are already in place. 
Both front-end (i.e., award processing) and back-end (i.e., audit processing of disburse- 
ment data) checks are currently in existence in Pell Grant processing. Front-end 
processing procedures exist to maintain some QC measures for the keying of 
application data by the application processors. The processor provides a basic data 
source for improved QC on student and institutional performance. Front-end QC 
checks are currently made on the SAR data in PIMS. After the SAR is produced for a 
student, the data on the form are verified and validated prior to determining tt e 
amount of Pell Grant funds to be disbursed to the student. In addition to front-end 
processing checks, there are back-end disbursement checks in PIMS. 

The Pell Grant Branch of DPO is also implementing a system for identifying 
error-prone institutions based on edit checks. The purpose of this QC subsystem would 
be to further systematize these processes so that the number of errors can be 
monitored and further reduced. 
Location of the Subsystem 

The proposed student/disbursement subsystem combines and analyzes data 
generated by the front-end application processing and the back-end disbursement 
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processing* This subsystem would help identify errors and allow for error profiles to 
be developed for both students and institutions. This effort would reduce the amount 
of time DPO staff spend reconciling input errors to the student subsystem of PIMS. It 
would also provide a structured process for routine reporting of QC procedures that 
have been or are being implemented in DPO. The logical location of this subsystem 
*ould be in DPO. 
Subsystem Description 

This subsystem design activity would provide technical assistance to the Pell 
Grant Branch, DPO, in the development of a student /disbursement subsystem. 

The information flow of the student/disbursement subsystem would include: 

# Inputs from PIMS giving the number and types of students and institutional 
errors identified and corrected in the front-end processing of institutional 
progress reports, according to the edit formats currently being developed. 
Refinement of QC measures and standards would be required as a part of 
the subsystem design. 

# A process for providing an automated routine for reporting on errors 
identified in PIMS and the disbursement process. (It is also possible that 
error-prone modeling could be used to improve the disbursement process.) 

# Outputs giving 

Reports to DC?R on error-prone institutions that merit review 

Routine reports to the Deputy Assistant Secretary on the types of 
errors corrected in the disbursement process and the resultant dollar 
savings 

Trend data on problem :ir~2* in the disbursement process as an input 
to the QCMIS 

Benefits to OSFA 

The establishment of a student/disbursement subsystem can provide procedures 
for routinely monitoring error and identifying areas for corrective action. The major 
benefits of such a QC subsystem are; 
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• Reporting of appropriate QC information on a timely basis so that 
corrective actions can be taken 

• Development of standards and measures required for monitoring student 
data and funds disbursement 

• Development and refinement of student and institution error-prone 
profiles , which can be used to anticipate error rather than react to 
problems 

A student/disbursement subsystem would result in more timely reporting on error- 
prone areas in the Pell program. When the student/disbursement subsystem is fully 
operational, it is possible that the subsystem would replace the QC subsystem for 
annual assessment of payment error (external QC). However at the present time, the 
student disbursement subsystem is the most difficult to conceptualize since many of 
these features have been implemented or are being implemented by DPO. Therefore it 
should perhaps be developed after other subsystems. 

PROCESSOR/VENDOR QC SUBSYSTEM 
Statement of the Problem 

The Pell Grant processors are private contractors who process Pell Grant 
applications. Students send application forms to the processing contractor, who 
performs data entry, edits the forms and sends a Student Aid Report to the student. 
The purposes of the processor/vendor QC subsystem are to reduce errors by the 
processors and to reduce processor costs. 
Existing Procedures 

The current central processor is the System Development Corporation (SDC) in 
Santa Monica, California. Multiple Data Entry (MDE) contractors are the College 
Scholarship Service (CSS) in New Jersey, the Pennsylvania Higher Education 
Assistance Agency (PHEAA) in Pennsylvania, and the American College Testing (ACT) 
Program in Iowa. Each of these organizations presently has a QC program in place, 
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but the degree of effectiveness of the programs varies considerably. For example, 
PHEAA has instituted an application serialization and tracking system which ties into 
its on-line data entry system and greatly reduces the opportunity for losing applica- 
tions. The other processors have not implemented such a procedure. 

Aside from lost applications, the major QC concern for MD£ processors is the 
accuracy of data entry. Stage One found overall error rates in data entry to be rather 
low. Approximately 1 data entry error was found for every 37 applications. It was 
also found that ACT, which uses marksense scanning, had an error rate four times 
higher than PHEAA and CSS, which use key entry and verification techniques. 

It is at the central processor, SDC, where a QC subsystem would be the most 
valuable. SDC has a QC plan in effect. It incorporates appropriate QC at these 
points: 

• SAR processing 

• Roster production 

• Reports and statistical tables 

• Correspondence 

m Telephone inquiry 

• Archives and storage 

• Privacy and security 

• Energy conservation 

• On-line data correction 

• Special handling 

• Program materials distribution 

• Microfiche and microfilm products 

• ADS (Alternate Disbursement System) 
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• Validation services 

In addition to QC performed at these group levels, SDC has formed an 
Independent QC Review Team (IRT). SDCs QC plan employs serialized forms, reports, 
schedules, and charts, and at least in theory seems reasonably comprehensive. But 
because of cost overruns it would seem that the plan does not do an adequate job of 
controlling duplication of effort and inefficiency. In addition, the Stage One report 
specifies certain improvements which ought to be made in Pell Grant processing at the 
processor level. Therefore, an overall QC system should incorporate: 

• Existing SDC procedures where appropriate 

• Cost-saving measures 

• Stage One recommendations where appropriate 

• On-site monitoring and audit of statistical reports 
Location of the Subsystem 

A QC system for the processor would supply information to OSFA on the internal 
operations of the processor and possibly on the applicants. It would be one part of an 
overall QC system and would serve as a link between the student and OSFA. The 
information supplied by the system will be evaluated by OSFA, and corrective actions 
would be taken when error levels were exceeded. It would probably be located in 
DPPD. 

Subsystem Description 

The processor QC vendor subsystem can be broken down into three parts: inputs, 
processes, and outputs. 

• The inputs to a processor /vendor QC subsystem would include the follow- 
ing: 

Statistical data on keypunch error rates 
Statistical data timeliness of 5ER production 
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Statistical data on phone inquiry completion 
Statistical data on overall production 
Statistical data on validation activities 

• Statistical data or: application rejection rates and reasons for 
rejection 

• In general, the QC system process would be analysis of the input data to 
determine if error rates we r e within specified ranges. This presuppose* the 
existence of error ranges, and, in fact, one of the first tasks in developing 
a QC system would be the establishment of acceptable ranges. 

• The outputs would be a series of reports corresponding most likely on a 
one-to-one basis to the inputs. The reports would be designed to highlight 
(flagging with asterisk, etc.) unacceptable error rates. 

At the present time, the Policy and Program Development Division is responsible 

for monitoring the processor contract, and it would logically be involved in any QC 

system which included the Pell Grant processor. Cooperation between this group, the 

Division of Program Operations (for the Alternate Disbursement System), and the 

Division of Quality Assurance would be required to ensure the proper functioning of a 

processor QC system. 

Benefits to OSFA 

The benefits to OSFA of such a processor/vendor subsystem would be twofold. 
First, the accuracy of the data in the Pell Grant system would be improved. Even 
though the processor does not contribute substantially to the overall error, there is 
sti.4 good reason to take steps to reduce error regardless of its source. Secondly, the 
efficiency of the system would be improved with a corresponding reduction in costs. 
OS FA GOALS AND OBJECTIVES MONITORING QC SUBSYSTEM 
Statement of the Problem 

Until recently OSFA has lacked a capacity to monitor the performance of the 
student leiivery system. The goals and objectives system currently being 
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implemented by the Assistant Deputy Secretary for Student Financial Assistance 
would potentially provide such a mechanism* Technical assistance could be provided 
during the Stage Two QC study to assist with the implementation of this system, as 
well as to perform analysis necessary to develop a system for monitoring the Pell 
Grant delivery system* 
Current Procedures 

During recent months the Deputy Assistant Secretary for Student Financial 
Assistance has been implementing a goals and objectives system for student aid 
delivery* The system includes identification of: 

• Goals (OSFA wide) 

• Objectives (to be performed by divisions) 

• Activities required to implement objectives {by branches within divisions) 

• Subactivities required to complete activities (by section within branches) 

• Steps required to implement subactivities (by unit and individuals in OSFA) 
The current flow of information in this process includes top-down specification 

of goals and objectives and bottom-up specification of steps required to implement 
these goals* The goals themselves include specifics about time frames for each 
activity, subactivity, and step* This system is currently being automated by the 
division of system design and development. 

Currently, the system provides a simple mechanism for monitoring whether or 
not OSFA goals and objectives are being performed on a timely basis* The system 
currently does not provide a mechanism for performance of specific programs, since 
the information is organized by division rather than program. The system has the 
potential for providing program specific information about the delivery of the Pell 
Grant and other student aid programs (as an input to the policy model). 
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Subsystem Description 

The primary purposes of the subsystem development activity would be to 
provide: 

• Technical support required to implement and monitor the OSFA goals and 
measure process 

• Analytic support needed to generate QC measures and standards for the 
Peil Grant delivery system 

The technical support would include programming and systems design of support 

for the Division of Systems Design and Development, necessary for implementing the 

OSFA goals and measures process* This will include: 

• Design and enhancement of information flow and reporting for the sub- 
system 

• Programming of sample reports for OSFA branches, divisions, and the 
Deputy Assistant Secretary 

The analytic support would include the additional programming, systems analysis, 

statistical analysis, and policy analysis required to provide a system for monitoring 

performance of the Pell delivery system (and of other student aid programs if 

considered a high priority in OSFA)* Specifically, this analysis might include (but not 

be limited to); 

• Analysis of the time frame for the overall system delivery and the flow of 
information between divisions and branches 

• Input-process-output analysis for critical program functions (e.g., applica- 
tion processing, disbursement) with development of performance standards 
and tolerance levels for critical objectives 

• Analysis of activities, performance, including specification of performance 
standards for cne-time and ongoing activities 

• Identification of critical path flows: 

The program delivery cycle 

The policy (planning, budgeting, and evaluation) cycle 
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Benefits to OSFA 

The major benefits of this technical support activity would be: 

# Timely implementation and use of the OSFA goals and objectives process 

# Improved use of the goals and measures system for the purposes of 
improved program management and QC 

POLICY-LEVEL QC MANAGEMENT INFORMATION SYSTEM 
Statement of the Problem 

The management and control of the Pell program require timely information 
between the various subunits within OSFA. The purposes of the QCMIS concept 
identified here are to provide: 

• A monitoring tool for OSFA management to assess current system per- 
formance 

• A model of the system structured to permit the simulation of system 
operations under various changes in both policy and organizational 
variables 

The Current Situation 

The nature of the Pell program components is one of time and information 
interdependency. OSFA branches with responsibility for one aspect of the Pell 
program can usually perform effectively when completion times by related 
components are known and maintained, or when adjustments to known completion 
times can be appropriately anticipated. At present a substantial volume of data is 
collected and used by OSFA branches and divisions. The data collected are frequently 
required by other divisions as secondary or abstracted data. This interdependency, 
and the timely completion of various activities required to collect and transmit this 
data, are crucial to subsequent decisions and program operations. 

The timing of activities and the information flow for OSFA are currently being 
integrated into the OSFA goals and objectives system, coordinated by the Division of 



4-1S 



SG 



Systems Design and Development. The relatedness of data and subsequent actions, 
while well known within divisions, is not currently well articulated across divisions in 
OSFA. The automated goals and objectives system being implemented by DSDD has 
the potential of increasing OSFA policymakers' knowledge about the interrelatedness 
of program information* The effect, for example, of delays in award-level approval 
may be manifested as higher student error rates due in part to the lack of response 
time to reflect changes in publications. Monitoring of division and branch activities is 
dependent upon timely information flow, which in the case of most prevalent errors 
requires matching of information flows from different system components over a 
consistent time period. 

The description of the relationship between components on a time/information 
flow basis must account for a multitude of possibilities. The development of a QC 
management information system would provide the tools to anticipate those contin- 
gencies disruptive to the timely flow of operations. 
Location 

The QCMIS would logically use data on goals through steps generated by DSDD 
as part of the newly created goals measures system. It would also use information 
generated from program regulations developed in the Division Policy and Program 
Development. The logical placement of the subsystem would be in one of these two 
divisions. The primary user of the system would be the Deputy Assistant Secretary of 
Student Financial Assistance. 
Description 

Each branch and division in OSFA provides input to the OSFA goals system. This 
data, along with data generated from other QC subsystems (described earlier), would 
provide the basic inputs to the QCMIS subsystem. This subsystem would provide a 
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procedure for monitoring performance of the Peii delivery system and the potential 
for correcting errors in the system. A description of the direct and indirect 
reiatedness of their flows, dimensioned by time of occurrence, is the objective of the 
QCMIS. As currently conceptualized the flow of information in the QCMIS would be 
as follows. 

• Inputs for the QCMIS would be generated routinely from each of the QC 
subsystems. As part of the OSFA goals and objectives monitoring 
subsystem, critical path flows would be generated for the policy and 
program delivery cycles. Routine performance reports from the OSFA sub- 
system would provide the basic input to the QCMIS. Reports from other 
subsystems on critical error-prone points would provide an additional input. 
Other program and economics inputs would be identified as needed. 

• The QCMIS process would be comprised of two interrelated computerized 
simulation models; 

One based on the critical path flow of the policy cycle 

One based on the critical path flow of the program delivery cycle 

# Outputs from the QCMIS will include: 

Early assessments of the effects and costs of deviations from the 
critical path for the policy cycle 

Early assessments of the effects and costs of deviations from the 
critical path for the program delivery cycle 

A d hoc analytic reports assessing the potential effects of corrective 
action options 

# Special features could be developed to interrelate the critical path concept 
with an error-prone modeling capability. Such features would be dependent 
on the implementation of the above mentioned subsystem so, as additional 
subsystems are developed, additional error-prone modeling capabilities 
could be added to the QCMIS. 

Benefits to OSFA 

The QCMIS would provide OSFA staff with three principal benefits related to 
policy assessmei,., selection of operational alternatives, and improved efficiency of 
the QC monitoring process. These benefits are: 




• Policy assessment through micro-simulation 

• Assessment of operational activities 

• Efficiency of QC monitoring 

First, the QCM1S would provide the facility for simulation of the Pell delivery 
system in order to detect the effects of actual or anticipated changes in policy 
variables as a result of legislative action, changes in program operations, and time 
delays in one aspect of program delivery. In addition, OSFA staff may wish to use the 
MIS model concept to simulate alternative policies and procedures before making final 
policy recommendations. The simulation will provide staff with the knowledge about 
the probable effects of policy or program changes on the delivery system performance, 
error rates, and monitoring functions. 

Second, in addition to the facility for simulation of policy variables, the QCMIS 
could provide OSFA staff a tool for the assessment of alternative operational 
strategies. Input to the QCMIS is based on subsystem data streams which are 
reflective of the delivery system structure. OSFA staff may wish to investigate the 
effects of alterations to the delivery system structure as it presently exists. 
Subsystems with the greatest dependency on timely data may not currently be 
responsible for its collection, and such lags in information flow, as may result, will be 
identifiable within the QCMIS. A simulated shift of structure to provide such a 
subsystem with immediate access to the data can be performed without disrupting 
current operations. 

Third, the QCMIS provides a mutlilevei description of activity within and 
between each division in OSFA with the responsibility for the Pell delivery system. 
These activities are expressed as information/time linkages which afford OSFA staff 
substantial flexibility in simulation policy and operational effects. The linkages also 
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provide a framework on which audit and QC assessments can be made. Quality control 
assessments expressed in terms of expectations for the Peii delivery system by branch 
3nd division can be readily derived from the QCMIS approach. Conforming to standard 
QC approaches, this aspect of the QCMIS can include system tolerance conditions, 
systems procedures monitoring, and error-prone profiling for the OSFA function. This 
would result in identification in tolerable error levels for each branch and division with 
the responsibility for Pell program delivery, and such an approach would provide OSFA 
staff a management device capable of producing performance expectancies for the 
system as a whole and for subsystems independently. An actual versus expected 
performance analysis generated by the QCMIS would provide OSFA with the most 
accurate and timely QC monitoring tool. 

OTHER STUDENT AID PROGRAMS 

It is possible that other student aid programs can be integrated into the modular 
design outlined above. Two types of design considerations would need to be considered 
in order to accommodate other programs: 

# Subsystems that could be modified to accommodate other programs 

« New subsystems for other programs 

In order to fully address thes^ design issues it would be necessary to conduct a 
functional analysis for the Campus-ased, Supplemental Educational Opportunity Grant 
(SEOG), College Work Study (CWS), National Direct Student Loan (NDSL), and 
Guaranteed Student Loan progrrms. However, in spite of this void, it is possible to 
speculate about how the other programs might be integrated into this framework. 
First, it is necessary to review some basic differences between the organization of the 
three program areas— Pell, Campus-Based, and GSL, Specifically, it is important to 
recognize: 
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• Egll ls 90 entitlement program that has a coherent set of information 
sources for institutions and students 

PIMS in DPO 

The processor contractor in DPPD 
The SMS system in DCPR 

• Campus-Based is not an entitlement program* Therefore it has a signif- 
icantly different allocation process, since institutions are directly respon- 
sible for award calculation and disbursement. There is no student data 
base for Campus-Based programs. 

• GSL ls also an entitlement program, like Pell, but it lacks the coherent 
data bases available for Pell. There is no student system, DCPR's lender 
reviews are less extensive than institutional reviews* 

In spite of these limitations, several of the subsystems discussed above could be 

modified or extended to include other programs* This might include: 

• Extension of the OSFA subsystem to include reports specific to all three 
programs 

• Modification of the institution review subsystem to include reports and 
analysis specific to ail three programs (the data are currently organized to 
make this a definite possibility) 

• Modification of the subsystem for annual assessment of overall payment 
error to include Campus-Based* (GSL would be impossible to measure in 
this framework, unless lenders and state agencies were included in the 
process.) 

• Extension of the QCM1S to include the other programs 

During the interviews, some additional critical error-prone areas were identified 
for GSL and Campus-Based programs. As a result, it is possible to identify some 
additional subsystems that would result in reduced error. Specifically: 

• An interest payment subsystem for a GSL to provide basic controls on this 
system 

• A FISAP subsystem for the allocation of Campus-Based funds 

While it is possible to extend the basic system design to include Campus-Based 
and GSL programs, it is important to recognize the importance of program specific 
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controls in the overall delivery system. A major conclusion of Chapter * was that at a 
managerial level in OSFA, it is essential that a QC system provide program specific 
QC measures and standards. The functional organizational structure in OSFA inhibits 
this development. When designing an overall QC system, program specifics and QC 
checks must be developed by divisions in OSFA. 

CONCLUSIONS 

The basic three-tiered information system perspective is useful in the design of 
an overall QC system for the Pell program. Each level of organization in OSFA 
requires different types of QC needs resources. In the functional analysis, three levels 
of QC requirements were identified for the Pell program: 

• At the policy level there is a need for analysis of 

Corrective action options 

Critical problems and their implications for the policy and delivery 
cycles 

Overall annual measures of payment error 

• At a program management level it is essential that program specific QC 
measures and standards be developed and monitored. 

• At the operations level, a series of ongoing QC issues were identified that 
can be addressed through systematic enhancement of existing information 
systems. 

Given existing QC requirements and information resources it is possible to 
propose a preliminary design that identifies modular QC components that address 
system needs at each of these levels. Accordingly, a five-component QC system was 
proposed. This included: 

• A policy level QC management information system to provide: 

Analysis of QC corrective action options 

Analysis of problems with the program delivery cycle 
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Analysis of problems with planning budget and evaluation cycle 

• An external QC subsystem for annual assessment of overall payment error 
in the Pell program 

• An QSFA QC goals and objectives subsystem as a source of critical path 
flow models for the policy cycle (planning, budget, and evaluation) and 
program delivery cycle for the Pell program 

• Student/disbursement QC subsystem for identifying error-prone institutions 
and student profiles based on analysis of application and disbursement data 

• Institution review subsystem for developing QC standards and measures for 
postsecondary institutions 

• Processor/vendor QC subsystem for monitoring performance of the Pell 
processing contractor 

It is possible to integrate student aid programs into this basic framework. Three 

of the subsystems— policy level QCMIS, OSFA QC, and institution/review can be 

modified to include both GSL and Campus-Based programs. The external QC also 

could be modified to include Campus-Based programs* Also, additional subsystems can 

be identified lor the disbursement function in the other programs. 

• An interest payment subsystem for GSL 

• A FI5AP QC subsystem for the Campus-Based fund allocation 

A critical issue when considering the specific design options desired, including 
where the basic design should be expanded to other programs, is that the QC system 
should facilitate program specific linkages between functions, through the develop- 
ment of QC measures and standards* The preliminary QC system design for the Pell 
program is sensitive to program requirements. If GSL and Campus-Based programs are 
added to this framework, these components should also be designed to respond to 
program specific requirements. 
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CHAPTER 5 

STRATEGIES FOR QC SYSTEM DEVELOPMENT 



The focus of this paper thus far has been on the design of a QC system for the 
Pell Grant Program, a preliminary version of which was presented in Chapter 4. 
According to the strategic approach to QC system development being used for this 
task, the next steps in the overall process are to: 

« Select and prioritize modular subsystems for development 

# Proceed with systems development of selected subsystems 

Before proceeding with these steps, however, it is necessary to consider two 
other issues related specifically to the strategies used to actual development of QC 
subsystems. Specifically, these issues are: 

• The technical approach to QC subsystem development 

• The development of a corrective action component of the overall QC 
system 

TECHNICAL APPROACH 

The successful use of the strategic modular approach to QC subsystem develop- 
ment requires that sound systems development methodology be used to develop 
modular subsystems* Due to the modular design and implementation strategy, the 
system design activity will require careful planning and coordination. Individual 
analysts will be responsible for the design of individual subsystems, while the task 
manager and Project Director will maintain responsibility for: 

• Designing an overall QC system with the necessary linkages between 
modular subsystems 

# Planning phased development of modular subsystems with necessary 
linkages between subsystems 

# Coordinating the activities of individual analysts with responsibility for the 
development of modular subsystems according to predefined steps 

5-1 
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QC System Design 

The overall syste.n Jesign will emphasize the development of a functional QC 
system that enhances processes and systems already in place in OSFA and utilizes 
existing information when possible. The QC system must provide QC analysis reports 
and information required at three levels: 

• Policy level : The overall QC system must provide the types of QC 
information, reporting, monitoring, and analysis required for improved 
policymaking. 

• Management Level : QC subsystems must combine to provide the types jf 
information tracking, monitoring, and analysis required to improve the 
quality of program delivery. 

• Operational Level : Individual QC subsystems must provide the technical 
and analytic support required to enhance the QC functions within division 
and branches within OSFA. 

When designing the QC system with modular components, it will be important to 

recognize that the strategic approach to QC system development will actually provide 

two services: 

• A predefined system development effort that uses a sound systems 
methodology 

• Technical assistance to OSFA divisions to enhance their operating systems. 
Phased Development of QC Modules 

Planning for the phased development of modular QC subsystems will involve a 
range of organizational, procedural, and systems considerations. First, the develop- 
ment of modular subsystems will require close collaboration between Advanced 
Technology and divisions and branches in OSFA, since divisions will have the 
responsibility for implementing and maintaining individual subsystems. This organiza- 
tional arrangement must be made initially by the Division of Quality Assurance, but 
the daily management of these activities will be the responsibility of the project and 
task managers. Planning for the compatibility between subsystems will be the 
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responsibility of the task manager and will require an assurance that the necessary 
linkages are built into each subsystem, which will be developed independently in a 
phased fashion. The arrangements made with each OSFA division must include the 
assurance that sound systems development methodology will be used. 

Second, procedures will be developed early in the project to allow for the phased 
development of individual modules. In some cases the development of modular 
subsystems can proceed independently. For example, development of a QC subsystem 
for monitoring overall payment error can proceed independently. In other cases, * 
phased development must recognize the interrelationship between subsystems. For 
example, development of the QCM1S will depend in part on implementation and 
refinement of the OSFA goals and objectives system. The phased development of 
subsystems recognizes these interdependences where they exist and requires a logical 
phased development of modules* Procedures used for subsystem development must 
provide for the review of linkages between systems during each step of the system 
development methodology. 

Finally, it is critical that a sound systems development methodology be used for 
the phased development of modular QC subsystems. Systems development 
methodologies typically includes six steps: 

# Survey of current systems, procedures, and resources 

# Functional analysis of operating requirements 

# System conceptualization (diagram actions and flows of information) 

# System design, with specifications for development (including programming 
for automated systems) 

# System development (including programming for automated systems) 

# System implementation (including actual implementation and testing of 
new procedures, programs, and processes) 
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To assure a technically sound approach to the QC subsystem development 
activity, these steps must be adapted to the modular approach used in this project. 
Steps in Subsystem D .velopment 

The subsystem development effort will have two purposes. First, it will provide 

technical support to OSFA divisions and possibly even the Office of the Assistant 

Deputy Secretary, for the refinement of existing subsystems. The implementation and 

maintenance of these enhancements will require involvement of Division personnel, as 

the design must be responsive to needs internal to the host division. Second, each 

modular QC subsystem should be an integral part of the overall QC system design and 

reporting system. This will retjuire strict adherence to a specified methodology. The 

steps in the system development methodology must serve both purposes. 

1.0 S ystems Survey * For sound systems development, each QC subsystem will need 
analysis of existing operating systems in the division as a basic input to the 
system design. To assure that necessary technical support is given the host 
OSFA division, the user's needs must also be surveyed- The system survey 
should: 

# Analyze functions of current system and subsystems in the host division 
(i.l) 

# Identify potential QC checks for the QC subsystems (1.2) 

# Determine areas for technical assistance related to QC (1.3) 

# Identify linkages to other QC subsystems (including QCMIS) (1 .k) 

2.0 Functional Analysis . The functional analysis is a detailed assessment of the 
organizational setting relative to the functions of the subsystem* At a minimum 
the functional analysis should: 

# Identify structural elements within the organizational hierarchy (within the 
division/branch and between divisions ) (2.1) 

# Identify the purpose of the QC subsystem (2 J 

# Determine the potential outputs of the QC subsystem 

3.0 QC Subsystem Conceptualization . After the results of the functional analysis 
are reviewed and approved by the task manager and OSFA Division personnel, 
the general conceptualization of the subsystem can be developed. With the 
functional analysis completed, the subsystem conceptualization should: 
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• Identify QC subsystem components (the flow of activity, procedures, and 
interfaces) (3.1) 

• Determine QC subsystem component activities (3.2) 

• Identify major QC subsystem interfaces both for manual and automated 
processes (including interface with other modular QC subsystems) (3.3) 

4.0 QC Subsystem Design . Based on the conceptual model of the QC subsystem, and 
any changes or modifications to the subsystem that result from reviews ^y OSFA 
personnel or the task manager, the subsystem design can be undertaken. This 
will include derailed specifications for: 

• Developing procedures for QC subsystem operations (4.1) 

• Identifying types of technical assistance required to implement the system 
(4.2) 

• Developing data acquisition and analysis plans (4.3) 

• Developing programming protocols (only automated QC subsystem) (4,4) 

5.0 QC Subsystem Development . When the user requirements and subsystem design 
are established work can proceed on the subsystem development. Prior to 
proceeding with QC subsystem development, DQA, the project manager, and the 
task manager should be assured that concerns about the overall QC system are 
addressed. Once these concerns are addressed, work can proceed on actual 
subsystem development, which will include: 

• Development of computer programs (as appropriate) (5.1) 

• Development of QC subsystem documentation (as appropriate) (5.2) 

• Conduct of appropriate analysis (including development of computer 
programs to translate or analyze data as an input to the subsystem) (5.3) 

6.0 QC Subsystem Implementation . Implementation of the subsystem will require 
direct collaboration between staff of the host division and Advanced Technology, 
Specifics on installation will vary from subsystem to subsystem. The installation 
process should: 

• Establish test plans and acceptance criteria (6.1) 

• Test the QC subsystem (6.2) 

• Train OSFA staff and others in QC subsystem operation (6.3) 

While the actual distribution of departmental needs (and corresponding levels of 
effort and types of technical assistance) will vary across targets, these steps represent 
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the foundation of our systems approach to problem areas. Within the steps for each 
target, specific tasks are identified, more to give flavor to our approach in each area 
than to provide a definitive statement of work. The appendix includes a systematic 
review of the following targets of opportunity: 

• Annual assessm *nt of overall oayment error in the Pell Gcant Program 

• Institutional QC— procedures and performance standards 

• Processor /vendor QC 

• OSFA goals and objectives management tracking system 

• Policy level QC management information* system (QCMIS) 

• FISAP process for institution allocations 

• GSL interest payment subsystem 

A target of opportunity has been specified for each of the subsystems discussed 
in Chapter except the student/disbursement subsystem. This was not included since 
many of the special features in such a system are currently being implemented. 
Additional targets of opportunity were developed for the F15AP allocation process and 
GSL interest payments. These were identified as critical errcr-prone areas in the 
OSFA interviews. 

r 

CORRECTIVE ACTIONS 

Another critical element in the overall design of a QC system is the strategy 
used to initiate corrective actions. This will require that attention be given tc the 
entire QC cycle, identified in Figure 1-1. 

While staff from Advanced Technology are working on the design of individual 
QC modules, the task manager will undertake the design of an overall QC monitoring 
system that considered the entire QC loop. The purpose of such a system would be to 
utilize information generated by QC subsystems to complete the QC cycle. The 
specific steps addressed in this design will be: 
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• Mobilize (or measurement. 

• Compare actual performance with standards, 

• Management decide corrective action needed. 

• Take corrective action. 

• Repeat cycle. 

At each set outlined above, a management decision is required in OSFA. The 
decisions will include: 

• Decisions about implementing QC modules (mobilizing for measurement 
based on cost/benefit analysis) 

• Decisions about performance that merits corrective actions based on 
analysis of QC data 

• Decisions on which corrective action to take (after analysis of policy and 
management options) 

• Decisions to implement, fund, and manage selected corrective action 

• Decisions to repeat or continue measurement in critical areas 

A systematic approach to completing the QC loop for critical error-prone areas 
will be developed during Stage Two. This will include consideration of: 

• New OSFA procedures required to implement the QC monitoring system 

• The formation of policy-level QC council, possibly headed by the Deputy 
Assistant Secretary 

• The types of policy analysis iequred to implement this system, including 
utilization of the policy level QCMIS 

Based on consideration of these and other factors identified by the Division of 

Quality Assurance and senior OSFA personnel, Advanced Technology could design an 

overall QC monitoring system. The specific content of this system would be identified 

as the Stage Two study continues. Some of the key activities necessary to develop an 

overall QC monitoring system are the following. 

• Develop Plan for QC Corrective Action System: Research and analyze 
alternative structures for an overall QC corrective action system in OSFA. 
Will include consideration of: 
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Formation of QC corrective action council, possibly headed by 
Deputy Assistant Secretary 

A corrective action reporting system using study results 

A system for developing and disseminating corrective action policies 
and procedures 

Develop QC corrective action decision framework: The effective develop- 
ment of a QC corrective action system will require detailed analysis of 
OSFA policy and management procedures, as well as Federal laws and 
regulations affecting the Pell Grant Programs. On the basis of such 
analysis, Advanced Technology will develop a corrective action decision 
framework that will include: 

Corrective action decision heriarchy that identifies corrective 
actions requiring new legislation, executive order, action by the 
Secretary of Education, as well as decisions that can be made 
internal to OSFA 

Corrective action decision criteria that relate types of action to 
degree of variance from specified measures 

Develop analytic plan for corrective action: In many instances corrective 
actions will require detailed analysis of policy and decision points. An 
analytic plan should provide: 

Decision criteria for analyzing policy options in specified areas for 
corrective action 

Analysis standards for corrective action analysis 

Report fort \. for corrective action analysis that will provide input 
with the Qv~ ^jrrective action system 



CONCLUSION 

An overall framework for developing a QC system for the Pell program has been 
identified in this paper. The overall objective of the paper has been to develop a 
preliminary design of a Pell program QC system. Based on a functional analysis, of 
the Pell delivery system, a modular QC system design was proposed. At this time it is 
critical that ED select and prioritize QC subsystems for development. An action plan 
has already been prepared by Advanced Technology on this topic and submitted to 
DQA for review. 
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TARGET OF OPPORTUNITY: Annual Assessment of Overall Payment 

Error in the Pell Grant Program 

Description of Problem(s) : 

• Object of QC* Overall Pell delivery system 

• Host* DQA 

1 .0 OSFA System Survey 

1.1 Analyze functions of current system 

• Determination of program error 

• Reporting of program error 

• Corrective actions 

1.2 Identify potential QC checks for QC subsystem 

• Measurement of overall program error 

• Error tolerance levels 

m Comparative performance standards 

1.3 Determine areas for technical assistance related to QC 

• QC sample design 

• Analysis plans 

• Corrective action planning 

1.4 Identify linkages to other QC subsystems 

• All other OSFA QC subsystems 

2 ,0 Functional Analysis of OSFA Subsystem 

2,1 Identify structural elements of the organizational 
hierarchy 

m Deputy Assistant Secretary 

• DQA 

• OMB 

2*2 Identify the purpose ( s) of the QC subsystem 

• Reduce fraud and abuse 

• Assure program integrity and intent 

• Identi f y opportunities for corrective actions 

2,3 Determine the potential outputs for the QC subsystem 

• Annual error report 

• Standards deviation report 

• Corrective actions target report 
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3.0 QC Subsystem Conceptual i ration 

3.1 Identify QC subsystem components 

• Data collection and analysis 

• Error detection and reporting 

• Corrective actions 

3.2 Determine subsystem component activities 

• Data collection and analysis: sampling, modeling 

• Error analysis and reporting: validation, 
measurement, evaluation, reporting 

• Corrective actions? short-term, long-term 
structural 

3.3 Major QC subsystem interfaces 

m Policy level QC MIS 

• Goals and objectives 

• Institutional 

4,0 QC Subsystem Design 

4.1 Develop procedures for QC subsystem operations 

• Data collection and analysis 

• Error detection and reporting 

• Instigation of corrective actions 

4.2 Identify types of technical assistance 

• Sampling 

• Modeling 

• Designing corrective actions 

4.3 Develop data acquisition and analysis plans 

m Application — reported, validated 

• Institutional records 

• Processor 

4.4 Develop programming protocols (automated only) 

• Structured programming requirements for system 

• Module Testing 

• Module compatibility testing procedures 
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5.0 QC Subsystem Development 

5.1 Develop computer programs (automated only) 

• Applicant sampling 

• Error detection and application validation 

• Modeling / related analysis 

5.2 Develop QC subsystem documentation 

• System users manual 

e Sampling and validation manual 

• Corrective action development procedures 

5.3 Conduct appropriate analyses 

• Error-prone \ 

• Validation methodologies 1 
m Trend 

6 .0 QC Subsystem Implementation 

6*1 Design test plans and acceptance criteria (automat 
only) 

m Operational tolerances review 

• Documentation compliance review 
m Validity of tolerance standards 

6.2 Test QC of subsystem 

• Validation, error detection 

• Error-prone modeling 

• Sampling 

6.3 Train OSFA staff, others in QC subsystem operation 

• Sampling 

m Error detection 

• Validation 

• Use of model 
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TARGET Of OPPORTUNITY t Institutional QC • Procedures and 

Standards 



Description: 



Object of QC: Institutions 

Host: Primary - DCPR, Secondary - DTD 



1 .0 OS FA System Survey 



1.1 Analyze functions of current system 

• Program review and audit information 
e Training materials 

• Institutional reporting and disbursement 

1.2 Identify potential QC checks for QC subsystem 

• Timeliness of reporting (institution and DCPR ) 
e Standards for award calculation 

• Standards for program management 

1.3 Determine areas for technical assistance related to QC 

e Assistance for analysis of SMS data 

• Training in QC procedures 

• Identifying error-prone institutions and 
corrective actions 

1.4 Identify linkages to other QC subsystems 

• Policy level QCMIS 

• Operations/disbursement 

• FISAP MIS 

2 .0 Functional Analysis of OSFA Subsystem 



2.1 Identify structural elements of the organizational 
hierarchy 

• DCPR structure and procedures 

• Other Divisions: DPO, DPPD, DTD 

• Deputy Assistant Secretary 

2.2 Identify the purpose ( s) of the QC subsystem 

• Establish institutional QC standards 

• Quantify institutional error 

• Plan for corrective action 

2.3 Determine the potential outputs for the QC subsystem 

• Institutional errci: reports (trends, institutions) 

• Exception reports to DPO 

• Corrective action procedures 



3.0 QC Subsystem Conceptualisation 

3.1 Identify QC subsystem components * 

e Institutional reporting and analysis 
e Disbursement by DPO 

e Institutional corrective action reporting 

3.2 Determine subsystem component activities 

e Reporting and analysis: identify error-prone 

institutions 
e Disbursement: report exceptions to DPO 
e Corrective actions: institutional self-correction 

3.3 Major QC subsystem interfaces 

e Internal: DCPR-SMS, regions 

e OSFA: QCMIS, goals and objectives, PIMS, FISAP 
e Institutional interface 

4.0 QC Subsystem Design 

4.1 Develop procedures for QC subsystem operations 

m SMS analysis and reporting 
* • Reconciliation 

• Corrective actions 

4.2 Identify types of technical assistanc e 

• Manual development 

• Training 

• SMS analytical support 

4.3 Develop data acquisition and analysis plans 
e Acquisition of SMS data 

e Institutional practices derived from case studies 

e Disbursement 

4.4 Develop programming protocols (automated only) 

e Structured programming requirements 
e Module testing 

e Module compatibility testing procedures 
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5.0 QC Subsystem Development 

5.1 Develop computer programs (automated only) 

• SMS data acquisition, analysis, reporting 

• Error-prone mode Una 

• Institutional corrective actions feedback 

5.2 Develop QC subsystem documentation 

• System user manual 

• Disbursement procedures manual 

• Multi-level institutional QC manual 

5.3 Conduct appropriate analyses 

• Exceptions 
e Trends 

• Corrective actions effects 
6 .0 QC Subsystem Implementation 

6.1 Design test plans and acceptance criteria uvttomated 
only) 

e Review of operational tolerances 

• Documentation compliance review 

• Validity of institutional QC standards 

6.2 Test of QC subsystem x 

• Reporting and analysis test for SMS 

• Error-prone and disbursement test 

• Corrective actions test 

6.3 Train (SB FA staff, others in QC subsystem operation 

• DCPR for SMS analysis system 

• DTD for corrective actions 

• DPO for disbursement 
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TARGET OP OPPORTUNITY i Processor/Vendor OC 
Description : 

# Chief of QC: Pell Grant Processor 

• Hosts DPPD 

1.0 OSFA System Survey 

1.1 Analyze functions of current system 

• Receive process applications 

• Generate SARs 

• Maintain and report applicant data 

1.2 Identify potential QC checks for QC subsystem 

• Existing contractor procedures 
m Cost saving measures 

• Stage One corrective action recommendations 

1.3 Determine areas for technical assistance related to QC 

• Operations analysis 

• Quality control procedures 

• Overall payment error 

1.4 Identify linkages to other QC subsystems 

• Policy level QCMIS 

• Goals and objectives 
m Overall payment: error 

2.0 Functional Analysis of OSFA Subsystem 

2.1 Identify structural elements of the organizational 
hierarchy 

• Structure and procedures DPPD, DPO 

• Deputy Assistant Secretary, Other OSFA divisions 

• External actors 

2.2 Identify the purpose ( s ) of the QC subsystem 

• Improve data accuracy 

• Improve efficiency and reduce processing costs 

• Develop SAR data base — for analysis and modeling 

2.3 Determine the potential output s for the QC subsystem 

• Exception reports 

• Cost reports 

• SAR data base 
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3.0 QC Subsystem Conceptualization 



3.1 Identify QC subsystem components 

• Sampling data analysis 

• Cost/ schedule control 

• SAR modeling 

3.2 Determine subsystem component activities 

• Sampling and analysis 

• Cost analysis/schedule implications 

• Participate projections 

3.3 Mp j^r QC subsystem interfaces 

• OSFA QCMIS, PIMS, goals and objectives 

• Internal contractor 

• Institutional interface 

4 .0 QC Subsystem Design 

/ 

4. Develop procedures for QC subsystem operations 

• Sampling 

^ Cost/schedule control 

• Modeling 

4.2 Identify types of technical assistance 

• Data analysis 

• Standards and measures development 
m Modeling 

4.3 Develop data acquisition and analysis plans 

• Production and cost data 

• Existing contractor QC plan and reports 

• Applicant data 

4.4 Develop programming protocols (automated only) 

• Structured programming requirements 

• Module testing 

• Module compatibility testing procedures 
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5*0 QC Subsystem Development 

5.1 Develop computer programs (automated only) 

• Sampling 

• Cost/schedule control 

• Modeling 

5*2 Develop QC subsystem documen ta t ion 

• System users manual 

• Technical systems documentation 

• Cost control procedures manual 

5.3 Conduct appropriate analyses 

• Error 

• Cost analysis schedule implication 

• Trends forecasting 

6 .0 QC Subsystem Implementation 

6.1 Design test plans and acceptance cri teria ( automated 
only) 

• Review of operational tolerances 

• Documentation compliance review 

• Validity of standards and measures 

6.2 Test QC of subsystem 

m Sampling test 

• Cost/ schedule control test 

• Contractor QC system test 

6.3 Train OSFA staff/ others in QC subsystem operation 

• Sampling 

m DTD DQA for cost/ schedule control, monitoring 
contractor QC procedures 

• DPPD modeling and analysis 
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TARGET OF OPPORTUNITY: OSFA Goals and Objectives Management 

Tracking System 



Description : 

• Object of QCs OSFA 

• Host: DSDD/Deputy Assistant Secretary for Student 

Financial Assistance 

1 ,0 OSFA Syste.n Survey 



1.1 Analyze functions of current system 

• Goals and objectives identification and tracking 
e Resource allocation 

• Organizational linkages identification 

1.2 Identify potential QC checks for QC subsyster 

• Early warning of critical problems 

• Performance monitoring 

• Opportunities for corrective actions 

1.3 Determine areas for technical assistance related to QC 

• System enhancement and implementation 
m Performance analysis 

• Corrective action planning 

1.4 Identify linkages to other QC subsystems 

• Policy level QCMIS 

• OSFA Divisions' reporting requirements 

• Corrective action reporting 

2.0 Functional Analysis of OSFA Subsystem 

2.1 Identify structural elements of the organizational 
hierarchy 

• Deputy Assistant Secretary 

• Division Directors 

• DSDD structure and procedures 

2.2 Identify the purpose ( s) of the QC subsystem 

• Track critical operations 

• Measure performance against goals and objectives 
m Identify corrective actions 

2.3 Determine the potential outputs for the QC subsystem 

« Early warning reports 

• Performance reports 

• Cqrrective actions 
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QC Subsystem Conceptualization 



3*1 Identify QC subsystem components 

• Critical area tracking 

• Performance measurement 

• Program analysis 

3.2 Determine subsystem component activities 

• Tracking critical areas* schedule slippages, etc 

• Performance measurement; Division reporting 

• Program analysis: analysis of delivery problems 
by program 

3.3 Major QC subsystem interfaces 

• QCMISx early warning tracking 

• Divisions* routine reports 

m Programs: corrective action analysis 

4,0 QC Subsystem Design 

4.1 Develop procedures for QC subsystem operations 

• Critical area tracking 

• Performance measuring 

• Program/corrective action analysis 

4.2 Identify types of technical assistance 

• System development/ enhancement 

• Policy analysis 

• Program/corrective action analysis and planning 

4.3 Develop data acquisition and analysis plans 

• Goals and Objectives Data 
m Performance Standards 

• Program Information 

4.4 Develop programming protocols (automated only) 

m Structured programming requirements for OSFA 
system 

• Module testing 

• Module compatibility testing procedures 
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5.0 QC Subsystem Development 



5.1 Develop computer programs (automated only) 

• Goals and object 4 -as 

• Early warning 

• Program analysis 

5.2 Develop QC subsystem documentation 

• System users manual 

• Early warning procedures manual 

• Program data analysis guide 

5.3 Conduct appropriate analyses 

• Critical point 
m Performance 

• Corrective action 

6 ,0 QC Subsystem Implementation 



6.1 Design test plans and acceptance criteria (automated 
only) 

m Review of operational tolerances 
e Documentation compliance review 
m Validity of program analysis guide 

6.2 Test QC of subsystem 

m Early warning test 

• Performance measuring test 

• Program analysis test 

6.3 Train OSPA staff, others in QC subsystem operation 

• n&rly warning system 

• Corrective actions 

• Program analysis 
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TARGET OF OPPORTUNITY: Policy Level QCMIS for Pell Grant Program 



Description 

• Object of QC: OSFA - All actors in Pell subsystems 

• Host: Deputy Assistant Secretary, Director, DPPD 

1 . 0 OS Fa System Survey 



1.1 Analyze functions of current system 

• OSFA goals and objectives 

• Program Laws, regulations, policies 

• Planning and evaluation 

1.2 Ide;.«.L£y potential QC checks for QC subsystem 

• Deviations from critical delivery schedules 

• Deviations from planning, budgeting and evaluation 
(policy) cycle 

• Measures of adequacy of internal resources 

1.3 Determine areas for technical assistance related to QC 

• Simulation modeling 

• Costing, forecasting analysis 

« Critical path scheduling and related analysis 

1.4 Identify linkages to other QC subsystems 

• Goals and objectives 

• Overall payment error 

• Other Pell subsystems 

2.0 Functional Analysis of OSFA Subsystem 



2.1 Identify structural elements of the organizational 
hierarchy 

• Deputy Assistant Secretary 

• Policy Level: 0MB, OPBE, Congress, Associations, 

• OSFA 

2.2 Identify the purpose (s) of the QC subsystem 

• Delivery cycle management 

• Policy cycle management 

• Simulate >>olicy and program options 

2.3 Determine the potential outputs for the QC subsystem 

• Delivery system performance reports 

• Critical performance criteria 

• Analysis of policy option's 
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3*0 QC Subsystem Conceptualization 



3*1 Identify QC subsystem components 

• Program delivery cycle monitor 

• Policy cycle monitor 
e Simulation model 

3.2 Determine subsystem component activities 

• Program cycles critical path analysis 
e Policy cycle: critical path analysis 

e Simulations assessment of policy options 

3*3 Major QC subsystem interfaces 

e Goals and objectives subsystem 
e All other Pell QC subsystems 

e Corrective action plans as input to other Pell 
subsystems 

4.0 QC Subsystem Design 



4.1 Develop procedures for QC subsystem operations 

0 Monitoring of program delivery cycle 
e Monitoring of policy cycle 
e Use of simulation model 

4.2 Identify types of technical assistance 

• Critical path scheduling 
e Performance analysis 

e Policy analysis 

4*3 Develop data acquisition and analysis plans 

e Program Data (laws, regulations , etc.) 
e Goals and objectives system report 
e Conditional variable inputs (program participation 
rates) 

4,4 Develop programming protocols (automated only) 

e Structured design requirements [ programming 
sys terns] 

# Structural design of critical path (program logic 
model) 

e Speci f ication of condi tional variables 
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5 ,0 QC Subsystem Development 



5.1 Develop computer programs (automated only) 

• Delivery monitoring 

• Policy monitoring 

• Simulation modeling 

5 • 2 Develop QC subsystem documentation 

• Monitoring procedures 

• Model operations manual 

• Policy analysis guide 

5.3 Conduct appropriate analyses 

• Critical path analyses 

• Relationship between delivery and policy cycles 

• Relationship among conditional variables 

6 .0 QC Subsystem Implementation 



6.1 Design test plans and acceptance criteria (automated 
only) 

• Review of operational tolerances 

• Documentation compliance review 

• Validity of systems and simulation output 

6.2 Test of QC subsystem 

• Delivery monitor 

• Policy monitor 

• Simulation model 

6.3 Train OSFA staff, others in QC subsystem operation 

• Program delivery monitoring 

• Policy monitoring 

• Simulation monitoring 
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TARGET OF OPPORTUNITY: FISAP Institutional Allocation Process 
Description : 

• Object of QC: Institutions 

• Host: Primary - DPO, Secondary DCPR, DTD 

1 .0 OSFA System Survey 

1.1 Analyze functions of current system 

• Application process 

• Processing of applications 

• Allocation of funds 

1.2 Identify potential QC checks for QC subsystem 

• Reasonableness standards 

• Standards for funds allocation 

• Standards for account reconciliation and close-out 

1.3 Determine areas for technic! assistance related to QC 

• Standards development and implementation 

• Error prone area analysis 

• Corrective action identification 

1.4 Identify linkages to other QC subsystems 

• Operations/disbursement 

• Policy level QCMIS 

• Institutional Subsystem 

Functional Analysis ,of OSFA Subsystem 

2.1 Identify structural elements of the organizational 
hierarchy 

• DPO structure and procedures 

• Other Divisions: DPPD, DTD, DCPR 

• External actors: NAP, contractor 

2.2 Identify the purpose(s) of the QC subsystem 

• Reduce application error 

• Enhance timeliness and effectiveness of allocation 
process 

• Control OSFA processing errors 

2.3 Determine the potential outputs for the QC subsystem 

• Deviation from reasonableness standards report 

• Processing error reports 

• Operational analysis file' 
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3 .0 QC Subsystem Conceptualization 

3.1 Identify QC subsystem components 

• Application review 

• Allocation process review 
m Reconciliation 

3.2 Determine subsystem component activities 

m Application^ reasonableness test, error analysis 

• Allocation i monitoring schedules , corrections , 
appeals 

• Reconciliation! compare FISCOP data to 
disbursements 

3.3 Major QC subsystem interfaces 

• Policy levels QCMIS 

• EDPMTS 

0 Institutional QC subsystem 
4*0 QC Subsystem Design 

4.1 Develop procedures for QC subsystem operations 

• Application review 

• Allocation process review 

• Reconciliation 

4.2 Identify types of technical assistance 

m Standards Development 

• Allocation procedures enhancement 

• Corrective actions analysis 

4.3 Develop data acquisition and analysis plans 

• Acquisition of FISAP data 

m Acquisition of institutional practices data 

• Program data; EDPMTS, formulae, regulations, etc, 

4.4 Develop programming protocols (automated only) 

• Structured programming requirements for FISAP 
system 

• Module testing 

• Module compatibility testing procedures 
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5 .0 QC Subsystem Development 

5.1 Develop computer programs (automated only) 

• Rea sonab lene s a te s t 

• Comparison of FISAP with EDPMTS data 

• Distributional analysis 

5.2 Develop QC subsystem documentation 

e Reasonableness test 
e Users manual 

• Reconciliation procedures 

5.3 Conduct appropriate analyses 

e Trends 

e Simulation 

e Distribution 

6 ,0 QC Subsystem Implementation 

6.1 Design test plans and acceptance criteria (automated 
only) 

e Review o2 operational tolerances 

m Documentation compliance review 

e Validity of reasonableness standards 

6.2 Test of QC subsystem 

• Reasonableness test 

e Test data matches between FISAP and EDPMTS 
6 - 3 Train OSFA staff, others in QC subsystem operation 
e Reasonableness 

• Comparison of FISAP and EDPMTS data 
e Distr ibvi t ional anal ys i s 
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TARGET OP OPPORTUNITY* GSL Interest Payment Subsystem 



Description : 

• Object c* QC: GSL lenders* processor 

• Host nr \^ 

1.0 OSFA System Survey 



1.1 Analyze functions of current system 

• Lender billing 

• Contractor processing 

• Disbursement/ EDPMTS 

1.2 Identify potential QC checks for QC subsystem 

• Inputsi Billing procedures, lender file 
, • Processes} General accounting 

• Outputs: Late payments multipayments 

1.3 Determine areas for technical assistance related to QC 

• Accounting procedures 

• Manual procedures 

• General QC procedures 

1.4 Identify linkages to other QC subsystems 

• Goals and objectives 

• Institution subsystem 

? 0 Functional Analysis of OSFA Subsystem 



2.1 Identify structural elements of the organizational 
hierarchy 

• DPO structure and procedures 

• Other ED - Deputy Assistant Secretary, Credit 

• Management Task Force 

• External Actors 

2.1 rdentify the purpose (s) of the QC subsystem 

• Assure accuracy of lender billing 

• Establish sound internal control 

• Establish control over payments 

2.3 Determine the potential outputs for the QC subsystem 

• Lender file update 

• Internal process reports [Error cross-check of 

• manual /automate payment files] 

• Summary payment report system 
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3 ,0 QC Subsystem Conceptualization 

3* 1 Identify QC subsystem components 
• Lender file review 

e Internal accounting/auditing procedures 
e Payment reconciliation 

3.2 Determine subsystem component activities 
e Lender t Update file 

e Internal i Cross-check interest billing with other 

GSL subsystems 
e Payment i Reconcile manual and automated payment 

procedures 

3.3 Major QC subsystem interfaces 
e Goals and objectives 

e Institutional 

e DCPR lender review 

4 .0 QC Subsystem Design 



4.1 Develop procedures for QC subsystem operations 
e Update 

e Prepayment audit and error correction 
e Reconciliation 

4.2 Identify types of technical assistance 
e Documentation 

e Accounting and auditing 
e Manual procedures 

4.3 Develop data acquisition and analysis plans 
e Lender file 

e Interest payments file 
e Disbursement tapes 

4.4 Develop programming protocols ( automated only) 
e Structured programming requirements 

e Module testing 

e DBMS schema analysis 

e Module compatibility testing procedures 



ERIC 



122 

A-20 



5,0 QC Subsystem Development 

5.1 Develop computer programs (automated only) 

• Lender update control 

m Interest billing prepayment 

• Reconciliation 

5.2 Develop QC subsystem documentation 

• System users manual 

• Technical systems documentation 

• Reconciliation procedures 

5.3 Conduct appropriate analyses 

• Currency of lender file 
0 Error 

• Trend analysis for account reconciliation 
6 . 0 QC Susystem Implementation 

6.1 Design test plans and acceptance criteria (automated 
only) 

• Review of operational tolerances 

• Documentation compliance review 

e Validity of reasonableness standards 

6.2 Test of QC subsystem 

• Update 

• Duplicate payment test 
o Reconciliation test 

6.3 Train OSFA staff, others in QC subsystem operation 

• Update of lender file 

• Prepayment audit and error correction 

• Reconciliation procedures 
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